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@ Improved phase comparator. 

@ Apparatus for transfer of data between de- 
vices having different dock frequencies in- 
cludes a first processor coupled to a first 
system bus to provide data to a cache and a 
memory, and a second processor coupled to 
the first system bus and a second abbreviated 
system bus to receive read data from said first 
system bus, and a means for conrecting errors 
In memory. Also included are a counter, means 
for detecting a selected state of said counter, 
and means, responsive to output signals from 
said counter, for selectively permitting or inhibi- 
ting transfer of data fed to a recirculating state 
device. The apparatus includes a first means for 
providing a first clocking signal, a second 
means for providing a second clocking signal, 
means for providing an error signal responsive 
to an offset between edges of the first and 
second docking signals. 
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This invention relates to phase connparators. 

The Invention is applicable to the field of fault tolerant computers and more specifically to the transfer of 
data between devices having different clock frequencies. 

As it is known in the art often a computing system is assembled from several functional units which op- 
5 erate together to perform a given task as quickly as possible. Often functional units which exchange data may 
be operating at different clocking frequencies, for example a Central Processing Unit (CPU) may operate at 
a faster clocking frequency, or cyde time, than an Input/Output (I/O) unit. 

In order to ensure thata valid data transfer occurs, a common frequency for the two functional units is generally 
derived. For example, if a given CPU unit has a cyde time of 40 ns, and a given I/O unit has a cyde time of 

10 50 ns, then the easiest obtainable cyde time is 200 ns. Once the common frequency has been selected, each 
of the two docks periods is multiplied by the appropriate number to provide two clock signals which are oper- 
ating at the same frequency. 

Although both the CPU and the I/O unit may be operating at the same frequency, they may not be phase 
matched, and data may not be accurately transferred between the two functional units. To ensure accurate 

15 data transfer, a common method of phase matching clocks is through the use of a phase locked loop. 

Typically in phase locked loop designs, one clocking signal is provided as a reference dock, and the relative 
phase of the second clocking signal with respect to the reference dock signal is adjusted until both dock signals 
are in phase. The phase locked loop allows the two docking signals to remain in phase during operat Ion despite 
variances in the reference docking signal due to temperature changes, etc, and thus data transfer between 

20 the two functional units is ensured throughout operation. 

The reference clod< signal on one functional unit is typically driven by a fixed oscillator. The adjusted clock- 
ing signal is typically provided by a variable frequency oscillator, for example a voltage controlled oscillator. 
As discussed previously, the two clock signals may have different docking frequencies, and therefore a phase 
locked loop may Include a dh^ider circuit for each dock signal to adjust each of the docking frequencies to a 

25 common frequency. 

A phase detector circuit is used in the phased lock loop to compare the phase of the reference clock signal 
to the phase of the adjusted clocking signal and to provide an error signal to adjust the phase of the voltage 
controlled osclllatDr. One type of phase detector is a three state phase detector vifhich detects a rising edge 
of each docking signal. Two signals are provided from the three state phase detector, one signal which is a 

30 reference clock detect signal and one signal which is the adjustable frequency detection signal. Each detection 
signal Is asserted when the respective dock signal is asserted, and remains asserted until both dock signals 
are asserted. Thus, each detedion signal is basically a train of pulses, with the width of each pulse in the train 
of pulses indicative of the phase difference between the two docking signals. The reference detection signal 
and the adjustable frequency detection signal are fed to a low pass filter to remove the reference dock signal. 

S5 The filter provides the error signal as a voltage level to the voltage controlled oscillator with the voltage level 
being related to the phase error between the reference dock signal and the adjustable dock signal. 

One problem with the above mentioned phase detector occurs when the common frequency which is pro- 
vMed has a relatively low frequency compared to the frequency of the reference and variable clock signals. 
As the common frequency decreases, the timing error, or skew, between synchronized clock edges increases. 

40 A low timing error, or skew, is desired between the corresponding pulses of the two dock signals to reduce 
noise in the clock signals. Lower noise levels permit logic designs having longer propagation delay chains to 
be provided between clock edges. The timing error between the dock signals is related to the characteristics 
of the phase detector. In particularthe timing error is related to the phase error as shown by the below equation: 

Te = ♦e/Fd 

45 where: 

Te is the timing error, or skew, between relative dock pulses 
4|>E is the phase error provided by the phase detector 
Fd is the frequency of the detection pulses provided by the phase detector 
One problem, as can be noted from the above equations and discussions of the origin of the detection 
so pulses, as the common frequency decreases, the timing error is increased due to the concomitant decreasing 
frequency of the detedion pulse train. Moreover, the reference frequency components of the detectbn pulse 
train from the phase detector drcuit should be filtered out prior to being provided to the voltage controlled os- 
cillator. As the reference frequency decreases the task of low pass filtering becomes more complex due to the 
effect of the filters phase response on the phase locked loop performance. Filtering of low frequency digital 
55 signals is generally difficult, requiring complex filtering circuits to remove the low frequency components. 

To overcome this problem, logic designers typically design logic circuits with fewer levels of logic provided 
between docked devices thus allowing for and compensating for higher levels of clock skew. This approach, 
while ensuring data transfer, is nonetheless undesirable since it typically increases the size of the logic circuit, 
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the number of synchronous levels in the circuit and hence the latency period to data transfer data through the 
circuit. 

The present invention is defined in claim 1 and preferred features are further defined in the claims de- 
pendent thereon. 

As described herein, a preferred embodiment of the inventive apparatus includes a first means for pro- 
viding a first clocking signal and a second means, responsive to an error signal, for providing a second clocking 
signal. Also provided is means, responsive to a rising edge and a falling edge of said first and second clocking 
signals, for providing the error signal. The error signal consequently adjusts the phase of the second clocking 
signal to provide a first and second clocking signals which are phase locked. 

The apparatus may include first detection means, responsive to said first clocking signal, for providing a 
first signal indicating a state transition of said first clocking signal, and a second detection means, responsive 
to said second docking signal, for providing a second signal indicating a state transition of said second clocking 
signal. Each of the first and second detection means, by detecting both the rising and the falling state transitfon 
of the clocking signals thus provides a detection signal with double the frequency of detection pulses as that 
provided by a typical three state phase detector device. This also doubles the phase detector gain around the 
lock point (i.e. zero degree phase error). 

The apparatus further includes means for deasserting said first signal and said second signal when said 
first signal is asserted and said second signal is asserted. Consequently, a pulse chain is created, with the 
pulse width related to the phase error between the rising edges or the falling edges of the first clocking signal 
and the second docking signal. The means for deasserting the first signal and the second signal further in- 
cludes means for preduding deassertion of said first signal and said second signal when said first signal and 
said second signal are both asserted due to opposite edges of said first and said second clocking signals. Thus, 
it is ensured that signals which are in fact 180« out of phase do not appear Id be in phase. 

By providing a pulse chain indk^ative of both the rising and the falling edges of the first and second clocking 
signal, the frequency of the detection pulses and the magnitude of the phase detector slope at the lock point 
is doubled, thereby reducting the timing error, or skew. Reduced skew permits increased levels of logic be- 
tween docked devices, and consequently the latency assodated with data transfer between two devices is 
reduced. Moreover, the relatively higherfrequency of the pulse chain provided by the apparatus fadlitates the 
design of a filter device used to remove the first docking signal components. 

A more detailed understanding of the inventton may be had from the following description of a preferred 
embodiment, given by way of example only and to be understood In conjunctton with the accompanying draw- 
ings wherein: 

Fig. lAisan Illustration of the physical cabinets containing a fault tolerant computer system In accordance 
with a preferred embodiment of the present invention; 

Fig. 1B is a block diagram of a dual zone fault tolerant computer housed in the physical cabinets of Fig. 
1A. 

Fig. 2 is a diagram showing the relationship between Figs. 2Aand 2B; 

Fig. 2A is a block diagram of a primary rail, phantom rail fault tolerant processor system used in each zone 
of the dual zone fault tolerant computer system of Fig. 1 A. 

Fig. 2B is a block diagram of a input output interface to the primary rail / phantom rail computer system 
of Fig. 2A. 

Fig. 2C is a block diagram of a network interface device used in the fault tolerant processor of Fig. 2; 
Fig. 2D is a flow chart showing the steps in assigning a system identity to the fault tolerant computer of 
Fig. 1A; 

Fig. 3A is a block diagram of a cache memory used in the primary rail / phantom rail computer system of 
Fig. 2A; 

Fig. 3B is a block diagram of a main memory for the primary rail/ phantom rail computer system of Fig, 2; 
Fig. 4 is a simplified block diagram of the primary rait / phantom rail computer system of Fig. 2A showing 
an associated input / output interface and bus interconnections for peripheral devices which is useful in 
understanding a technique for maintaining system identity in the feult tolerant computer system of Fig. 1 A; 
Fig. 5A and 5B are flow charts useful In understanding the technique for maintaining system identity as 
discussed in conjunction with Fig. 4; 

Fig. 6A is a block diagram of a bus Interface and control circuit used in the computer system of Fig 2A; 
Fig. 6B is a simplified block diagram showing the data flow path for the Bus interface and control circuit 
for Fig. 6A as well as a portion of a direct memory access controller interface In the circuit of Fig. 6A: 
Fig. 6C is a block diagram of an error correction / detection drcuit disposed in the data path of Fig. 6B; 
Fig. 6D is a block diagram of a parity checker used in the data path circuit of Fig 6B; 
Fig. 7 is a block diagram showing dock synchronization between the pair of zones of the fault tolerant com- 
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puler system of Fig. 1A; 

Fig. 8 is a simplified block diagram of the primary rail phantom rail computer system of Fig. 2 showing 
clock circuit distribution and clock alignment circuits for the computer system of Fig. 2; 
Fig. 8Als a timing diagram showing various timing relationship of signals illustrated In Fig. 8; 
5 Fig. 9 is a block diagram of a clock interface circuit used to Interface a system dock to an I/O dock for use 
in the computer system of Fig. 2; 

Fig. 9A Is a timing diagram showing the timing relationships between signals described in conjunction with 
Fig. 9. 

Fig. 10 is a logic diagram of a frequency divider circuit used in the dock interface circuit of Fig. 9; 

10 Fig. 11 is a diagram of a enable selectbn circuit used In the dock interface circuit of Fig. 9; 

Fig. 12A is a logic diagram of a recirculating state device for transferring data synchronized to a CPU dock 
to data synchronized to an I/O dock used In the dock interface circuit of Fig. 9; 
Fig. 12B is a logic diagram of a recirculating state device for transferring data synchronized to an I/O dock 
to data synchronized to a CPU dock used in the dock interface circuit of Fig. 9; 

15 Fig. 1 2C is a logic diagram of a recirculating state device for transfering level signals between an I/O dock 
and a CPU dock; 

Fig. 13 Is a logic diagram of a pulse width nrK)dulator circuit used in the dock synchronizer circuit of Fig. 
8; 

Fig. 14 Is a block diagram of a phased locked loop for aligning docks between a CPU dock and an I/O 
20 dock used in the dock interfece circuit of Fig. 9; 

Figs. 15A, 15B, and 15C are transfer curves Illustrating the relationship between the phase error and out- 
put voltage of a prior art phase detector and the phase detector of the present invention as shown in Figure 
14; 

Fig. 16 is a schematic diagram of a phase detector circuit used in the dock interface circuit of Fig. 9; and 
25 Figs. 17A -17C are timing diagrams illustrating the alignment of clocks by the phase detector drcuit of 
Fig. 16. 

Referring now to Figure 1 , a fault tolerant computer system 1 0 embodying the present invention includes 
duplicate processing systems 12/12' physically shown here in a rack mounting system 11. The two processing 
systems 12/12\ called zones, are physically separated by support bar 3. Communication cables 9 enable conrv 
30 munication between the two zones. 

Zone 12* indudes separate elements corresponding to elements of zone 12 so, although only zone 12 will 
be described in detail, it is to be understood throughout this specification that identical elements performing 
identical functions are Induded within zone 12'. Zone 12 includes power supply 4. disk drives 8, and functional 
units such as Central Processing Unit (CPU) and Memory Unit 15, I/O unit 25, network interface unit 30, and 
35 control panel 19. 

The control panel 19 indudes an Non-Volatile Ram (NVRAM) 20 which contains a system identification 
number for the entire dual-zone system. Since a common application of a fault tolerant computer 1 3 is as one 
computer system on a network of computer system or other types of computer devices, a network controller 
in general needs to know an identity for each system attached to the network. Thus, a fault tolerant computer 

40 13 has a unique network address or system identification number. A derivative of the system identification 
number is copied to the network interface unit 30 and stored in Non-volatile memory to be used as the network 
address of the dual-zone system 10. 

As explained in greater detail below, each zone 12/12' includes several modules interconnected by back- 
planes. If there is a fault or error in a unit, the unit may be removed and replaced without disabling the computing 

ds system 1 0. This is because, although zones 12, 12' are physically housed within the same cabinet, each zone 
12,12' has its own components and is capable of operating Independently of the other zone 12M2. Thus, com- 
ponents within one zone may be replaced while the other zone continues to operate. 

Referring now to Figure 1B, the dual-zone processing system 10 of Figure 1 A indudes two CPU/Memory 
Units 15/15' coupled by resynchronization bus 15e. The resynchronizatlon bus 15e is used to ensure that the 

so contents of a memory subsystem 50 of the CPU/Memory unit 1 5 are consistent with the contents of memory 
50' in CPU/Mem unit 1 5'. For example, when one zone is taken out of the dual-zone system 1 0 for repair, and 
a new one is Installed into the system, the contents of the memory in the remaining zone must be copied to 
the newly installed zone. The 32 bit bi-directional resynchronization bus 15e provides a convenient pathway 
for memory data exchange. 

55 Each of the CPU/Memory units 15/1 5' is coupled to a respective t/0 unit 25/25'. The I/O unit provides an 
interface between the CPU/Memory units 15/15' and a variety of external devices, such as a disk drive, a con- 
sole, or a network. Each of the I/O units 25/25' Is coupled to the respective CPU/Memory units 15/15' by 16 
bit I/O bus. 15c/15c'. In addition, busses 15d/15d'. which are exact copies of busses 15c/15c', also provide an 
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interface between CPU/Memory units 15/15* and I/O units 25/25\ Providing an exact, or "phantom" copy of 
the busses 15c/15c' across the interface between the CPU/l^emory unit and the I/O unit allows for checking 
of the data integrity across the unit Interface. 

ROM busses 15b/15b' couple each of the CPU/Memory units 15/15' respectively, to a flash ROM (Read 
5 Only Memory) 18a/18a' and a set of error registers 18b/18b' on the I/O units 25/25' as well as an error register 
17 on the CPlAMemory unit 15/15'. 

The I/O units 25/25' are coupled to a variety of external devices by 32 bit busses 25d/25d*. Each of the 
busses 25d/25d' may support up to 8 external devices, referred to as 'personality cards', off of a port connec- 
tions (personality ports). By way of example, a Digital Small Systems Interface (DSSI) ™ of Digital Equipment 

10 Corporation personality card 29 and a Network Interface (Nl) personality card 30 are shown in Figure IB. 

Additional I/O unit 26 may be coupled to I/O units 25/25' of the dual-zone system by Memory Interface 
busses 25a/25a'. The addltbnal I/O unit 26 may also be an I/O interface unit, supporting an additional 8 per- 
sonality cards in each zone, allowing for 16 personality cards In each zone, or a total of 32 personality cards 
(I/O device interfaces) in each dual-zone system 10. 

15 Each of thefuncttonal units, including the CPU/Memory Units 15/15', the I/O units 25/25', the personality 
cards, and the control panel19/19', includes a Non-Volatile RAM for storing board information about each unit 
such as manufacturing revision number, serial number, or and network address. A set of serial Inter-Integrated 
Circuit (|2C) busses 1 5a/1 5a', 25c/25c' , and 25d/25d' allows for access to these RAM devices by the CPU/Mem- 
ory units 15/15'. The RC bus protocol is defined by Philips™ Corporation. 

20 A system PC bus 15a/15a' couples the NVRAMs 20/20' on control panels 19/19' to the respective 
NVRAMs 14/14' in the CPU/Memory units 15/15', the respective NVRAMS 24/24' in the I/O units 25/25' and 
the respective NVRAMs 26a/26a' in any additional I/O units 26/26' to a system PC controller 21/21 ' on I/O units 
25/25'. 

A personality card (PCARD) PC bus 26d/25d' couples all of the personality cards on a given bus to an TO 
25 controller on the I/O unit which is driving that bus. (For example, PCARD PC bus 25d/25d' couples the 
NVRAMS on DSSI pcard 29/29', Nl pcard 30/30', and other pcards 31/31' to an PC controller 23/23' on I/O unit 
25/25*. However, any PCARDS driven by I/O unit 26/26' would be coupled to a PCARD I^C bus 26c/26c' which 
would be controlled by an |2C bus controller 26b/26b' in unit 26/26'), 

A POWER |2C bus 26C/25C' couples a power supply controller (not shown) to a third |2C bus controller 
30 22/22' In the I/O unit 25/25'. Each of the I^C bus controllers (21/21 ',22/22' and 23/23') are available to 
CPU/Memory units 25/25' to allow the system to read and write data to the NVRAMS, and to monitor and control 
the power supply. 

A parallel bus 25f hereinafter referred to as a crosslink bus couples the I/O units 25/25' of each zone 12/12'. 
Each time the dual-zone processing system performs an operation which accesses I/O space, checks are per- 

35 formed by hardware in each of the I/O units to ensure that both zones are performing the same I/O access. 
One zone drives the crosslink bus, while the second zone checks the data on the bus to ensure that it is per- 
forming an identical I/O task. That is the zone which requests the bus is the zone which drives the bus while 
the other zone checks the data driven on the crosslink bus. If there Is a miscompare In the I/O requests between 
the two zones, the dual-zone system has fallen out of lock step, and a system error is logged. 

^ A serial crosslink bus 25g also couples the I/O units 25/25' of each zone 12/12'. The serial crosslink bus 

25g is used during re-synchronization to enable the two zones to communicate before they have achieved lock 
step operation. The use of the serial crosslink bus 25g during Initialization and re-synchronization will be dis- 
cussed In more detail later In the specification. 

45 Fault Tolerant Philosophy 

The synchronization of the two zones 12/12* Is implemented by treating each system as a deterministic 
machine which, upon receipt of the same inputs and starting in the same known state, will always enter the 
same machine state and produce the same results unless there Is some error. Zones 12/12' are configured 
so identically, receive the same inputs, and therefore pass through the same states. Thus, as long as both proc- 
essors operate synchronously, they should produce the same results and enter the same state. If the proc- 
essing systems are not in the same state or produce different results, it is assumed that one of the zones 
12/12' has faulted. The source of the fault must then be isolated In order to take corrective action, such as 
disabling the faulting unit 

55 Error detection generally involves overhead in the form of additional processing time and hardware. To 
minimize error detecting overhead, a system should check for errors as Infrequently as possible consistent 
with fault tolerant operation. Fault tolerant operation necessitates the ability to detect an error in a zone, the 
ability to correct the error, and the ability to resume operation without impacting the processing of the computer 
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system 10. At a minimum, however, error detection must occur before data from CPU units 15/15' propagates 
to the external systems. Otherwise, corrupted CPU data may cause Improperoperation of the external system, 
with unpredictable consequences. 

Additional error detection hardware Is desirable in order to Isolate the fault within the CPU/Memory unit 

5 15. For example, If the fault can be Isolated to one of the components on the CPU/Memory unit 15. the com- 
ponent may be removed and replaced, thereby saving the cost of replacing the entire unit. Each zone of the 
present invention, which combines hardware redundancy with error correction codes to optimize error detec- 
tion and fault isolation on the CPU/Memory unit 15, as will be discussed below. 

Referring now to Fig. 2, Figs 2A, and 2B, a fault tolerant, single fall stop computer system 13 is shown to 

10 include a so called primary rail computer 41 and a phantom rail computer 4V. Each of the zones 12, 12' (I.e. 
zone A, zone B) of the fault tolerant computer system (Fig. 1 A) would include one of said single fall stop com- 
puter systems 13. The primary rail 41 and phantom rail 41' are two computer systems which are disposed to 
operate in so called "lock step" operation as will be described below and which are used to provide a certain 
degree of computing redundancy and thus fault tolerance to the computer system 10. The primary rail 41 is 

f5 substantially identical to the phantom rail 41' and thus only the differences in the phantom rail 41' with respect 
to the primarily rail 41 will be discussed below and a complete discussion of the primary rail 41 will suffice 
for a discussion of both the primary rail 41 and the phantom rail 41'. 

The primary rail 41 is shown to include a central processor unit 42, here any standard processor or mi- 
croprocessor type of device. A preferred example for processor 42 Is the so called NVAX-f- chip manufactured 

20 by Digital Equipment Corporation as part number 21-34684-01. Alternative chips include 64 bit processors 
such as Digital part number 21 064. Here, the NVAX+ CPU Is used for the central processing unit 42. 

The primary rail computer system 41 further includes a system Bus 43 comprised of address Bus lines 
43a. data Bus lines 43b, ECC Bus lines 43c and control Bus lines 43d respectively as shown. The system Bus 
43 has one port which is coupled to respective ports of the CPU 42 and has a second port which is coupled 

25 to a memory cache 46. Memory cache 46 will be further described in conjunction with Fig. 3A. Suffice it here 
to say however that memory cache 46 is a shared memory resource between the primary rail computer 41 
and phantom rail computer 41 ' as will be further described. A second port of the system Bus 43 is coupled to 
a Bus mierface circuit 54. Bus interface circuit 54 here comprises an address interface circuit 54a and a data 
interface drcuit 54b. Details of construction of the bus interface circuit 54 will be discussed in conjunction with 

30 Fig. 3B. 

The output from the Bus interface circuit 54 are a pair of Busses 55, 56. A first Bus 55, the memory Bus 
has address, data, and check bit lines as well as control signal line between the Bus interface circuit 54 and 
a main memory 50. Main memory 50 will be discussed with Fig. 3B. Suffice it here to say that main memory 
50 is a resource which is shared in common between the primary rail 41 and phantom rail 41'. The primary 

35 reil 41 has a second Bus here an internal interface Bus 56 comprised of address and control lines which are 
fed between the Bus interface circuit 54 and a Bus interface / control circuit 60. Bus interface circuit 54 as well 
as Bus interface and control circuit 60 are provided to format data from the system Bus 43 to the reminder of 
the primary rail 41. That Is the system Bus 43 has typically a relatively wide data Bus here for example 128- 
bits of data and correspondingly 28-bits of error correcting code associated with 128-bits of data. The Bus 

40 interface circuit 54 formats the data into a data width of here 32-bits and corresponding ECC bits into a width 
of 7 bits which accommodates the width of the data and the ECC bits associated with the internal interface 
bus 56. The Bus interface 54 also contains part of the control logic for a direct memory access operation in 
the primary rail 41. Likewise, the data interface circuits 54b-54e format the data into an appropriate width of 
here 256 data bits and 56 ECC bits for the memory. 

45 The Bus interface and control circuit 60 is fed by the Bus 56 and provides outputs on a Bus 15c. Bus 15c 

is 16-bits wide. Thus, the Bus interface control circuit 60 formats the data from the Bus 56 to the appropriate 
width for the Bus 15c. The Bus interface / control circuit 60 further includes control logic to control a DMA op- 
eration in the primary rail 41, interfaces to the resynchronized Bus 15e which is used to resynchronlzed the 
memory in zone 12 with the memory in zone 12' or vice-versa during a system resynchronization and further 

so interfaces to a read only memory (ROM) 18a via the ROM Bus 15b (Fig. IB). Further details of operation of 
interface and control circuit 60 will be discussed in conjunction with Figs. 6A-6C. 

ROM Bus 15b also communicates with error register 17 on the CPUVMem unit 15. Error register 17 is used 
to store errors of both the primary and phantom rails associated with accessing memory unit 50. These errors 
include single bit memory errors, double bit memory errors, ECC errors between the primary and phantom 

S5 rails as well as address and control errors between the primary and phantom rails. Details associated with 
accessing the memory unit 50 will be further discussed in conjunction with figure 3B. 

The phantom rait 41' has components similar to those in the primary rail 41. The phantom rail 41' includes 
a CPU 42' having a system Bus 43 including address signals on address Bus 43a', data on data Bus 43b', error 

7 
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?«?f °" • "'8"^'^ °" 43d'. Data bus 43b is coupled via a 

nLl^lT /J! T"?,?."*^ *° ^ ^^"^"'^y CPU 42'. The standard driver 45' M cou- 

ples to data bus43b to bus 43b provides improved signal integrity of the data bus 43b'. The CPU 42/42' which 

oJTl^aJ/! n 1!?''°? '^^^ ^^'^ ^3b/43b' against the ECC bits received 

on Dus 43C/43C on I/O cycle or memory read cycle. 

41' Rathr,"„"il]?„r"°" °? '^^^ "'"'"9hout the remainder of the phantom rail 

c^^ ! ""^ ^"^ ^ P^^"*""" «>"P'« CPU 42' to the cache 46. The 

!ilT.„ ? / ^e^P^-^ively are coupled to the cache 46 (Fig. 3A) as 

S id.^.Li"' 54'. The Bus interface circuit 54' also includes an address inte^ce ciLt 

2b' whlhJh'" *° ^'^^'^'^ '"terface 54a of the primary rail 41 and a data Interface cireui 

54^wh.ch .s here simpler m construction than the corresponding circuit 54b in the primary mil. Data interface 
circuit 54b IS here formats data associated with check bits on ECC Bus line 43c' 

drest nrr5?^t?pr"A'"*''^r.""L"^^ ^' ^"^"^' ^ "^"^^ ^5' here comprised of ad- 
srwhZ, rnnnlS K . !f o ""^ ^ P^^^^m mil internal Interface Bus 

S^ntTriiS^L H »K ^ ^ ^' '"'f ™' P''^*'^" °f 56c' Is also fed to Bus Interface 

?J?ir,h^ f ! ^ corresponding error correcting code portion 56c of Bus 56 in the primary rail is 

SeS' In cZtf'" "^''^ '""^'^ ™«^' Bus interlace Lul s 

!,Ti„?n f 1 . '"^ *° «^«'«'"""« that the processors 42/42' are oper- 

with out any errors in either of the two rails. 

The Bus interface and control circuit 60' is also fed data via the data Bus 56b of Bus 56 of the primary rail 

fJ^rf^jT^T"': T '""^^'"^ '"^ "'^ ^•"P"'^^ 20 through either oneTSe Bus 

interface and control circuits 60/60*. 

Buse^slS^^n'd rj^i^Jr' f^**' ^"J" *° P™^'^'"9 P~P^^ ^"^'"^^ ^-^^^ tf-e 32-bit wide 

fnS^i i!! ?K '"""^ ^"'^^ '"^"''^^ « '^'^'^ '"'«rfa'=« '''^•^"it 90 to provide clocking 

interface be ween the system CPU dock operating the CPU's 42/42' and an Input / output (I/O) dock. dSs 

^ he^r^ 1 "'f«?°'*. '"''^'^^ """^ '° «>"j"nctk,n With R^. 9-1 2 sSc^ 

It here to say that neither the frequency, nor phase used to clock CPUs, 42/42' need to be the ^me aTtS 

clock used With the Input/output(l/0)unltscoupled to the busses15c/15c as wo^^^ 

° '"^'y^^ '^^^ on the bus 150/15C' between two systemS operatC^h diS 
frequency and phase relationships. » wiui umereni 

92h«rfrT°'^' ^"^ andcontrol circuit eOfurtherlndudesaDMAoontroller 92. DMAcontroHer 

le« gSSl bofh Jtir T. "'"t^' the phantom rail 41 '. The DIVUV control 

larae w!Lrcf f IT T P^«=^°^« *2/42' from controlling the transfer or moving 

h?sin!S^tf ? ?' particularly during raquests by I/O devices that may be attached to 

L„2rV«i if "^'"''^ "^""^ the art for fault toleram 

computers as well as computer systems in general. Although most of the DMAcontroHer is contained in Bus 

descnbed in conjunctton with Figs. 8-6E. 



Referring now to Figure 2B. associated with both the primary rail 41 and the phantom rail 41' are com- 
munication devices 65/65' used to exchange data between the tie zones and here riZd to aL 
communicat^n devices coupled to bus interface and control circuits 60/60' by 16 bit busses 15c and^ t 
s^^ively. The 16 bit busses 15cand 15d indude identical data, bus 15c reflecting the proissin^ of the pt 

Ses-Trldrilntl ''i "^T "^'^"^ P'^"*°" '^'^ communicatL deviSL 

6»e5 indude interfece logic to control the interface of the CPU devices 42/42' and memory 50 to a plurality 
of I/O devices. In addition, the crosslink communication devices 65«5' indude logic to control the interfeS of 
CpS 'Z^Z ' w""' Via crosslink cables 25f and 25g', thereby ent^g he 

drJii^/Ss Zri'°"f " ""T"* '° '''^'^ ''^^^ The crosslink communication 

devices 65/65 also function to ensure that CPUs in both zones receive the same data from the I/O devices 
a the same time, thereby allowing the zones 12/12' to remain in lode step. The crosslink dmun ^tt^T 

ooSrhel 1"'*'"' »^ asslgnee'of the present iZZnTnd'^^" 

porated herein by reference. However other devices with similar characteristics may also be used 

Each one of the crosslink commun»ation devices 65/65' is coupled to a respective I/O bus controller chip 
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(FIREWALL) 70/70' by 16 bit Module Interface (Ml) busses 25a and 25b. Mi busses 25a and 25b each Inciude 
identical data reflecting processing by the primary and phantom rail respectively. The I/O bus controller de- 
vices 44/44' control the interface between the 16 bit Mi busses 25a and 25b to the 32 bit busses 71 and 72. 
In addition, the I/O bus controller device handles various system interrupts and I/O interrupts. The I/O bus 
5 controller devices 70/70' are also described in the above mentioned patent. However other devices with similar 
characteristics may also be used. 

Bus 72 couples the I/O bus controller devices 70/70' to a plurality of I/O control devices 73-78. These de- 
vices include a System Support Device (BSC) 73 which provides the system console port thus enabling con- 
sole communication with each zone. The console port of both zones 12/12' are tied together, and software 
10 driving these ports treats them as one logical entity. A processor device 74 and RAM device 76 (controlled 
by RAM controller 75) operate in conjunction with the SSC device 73 to execute console instructions. The SSC 
device 73 is any standard console device. 

The RAM device 76 comprises 4 Mbytes of dynamic memory. The read, write and refresh control signals 
for RAM device 76 are controlled by a memory controller device 75. The memory control device is a VAX mem- 
15 ory controller device designed by Digital Equipment company, assignee of the present invention, however 
other, similar devices may also be used. 

In addition to processing various console commands from the SSC device 73, the processor device 74 
operates In conjunction with the RAM device 76 as follows and an interrupt controller chip 77 as follows: when 
various I/O devices coupled to the personality ports of the I/O units 25 require action from the processors 42, 
20 the device generate an interrupt signal, which is collected by the interrupt controller chip 77, and fed to the 
I/O bus controller chips 70 which prioritize the interrupts and pass the interrupt onto the processors 42 through 
the busses 15c , 56 and 43. 

A transceiver 78 operates to convert the 32 bit bus 72 into a byte wide bus 78a for communication with a 
power controller (not shown) via I^C bus controller 21, bus controllers 22, and 23, a flash ROM 79 and a Non 
25 Volatile RAM (NVRAM) 24. Flash ROM 79 Is also a NVRAM which contains code to control the startup of the 
processor 74 and diagnostic code to test the integrity of the I/O connections to the 8 I/O devices 29, 30, 31 , 
32 (and an additional 4 I/O not shown) as well as to verify the functionality of various registers and memory 
locations within the processor device 74, SSC device 73, VIC device 77, memory controller 75 and RAM 76. 
The NVRAM 24 contains unit specific information such as serial number, revision number, and other manu- 
30 facturer specific Information. 

Bus controllers 21, 22 and 23 provide an Interface of serial Inter-Integrated Circuit (I^C) busses 15a, 25c, 
and 25d to the byte wide bus 78a. Here, the 1^ busses 1 5a, 25c, and 25d have two wires: a dock wire and a 
data wire. 

Bus 71 interfaces the I/O bus controller devices 70/70' to up to 8 I/O devices, (4 of which are shown) such 
35 as a DSSI device, an Ethernet device, a SCSI device, etc. Each of these devices 29, 30, 31 , and 32 are herein 
referred to as personality cards, and the connection port for a personality card to the bus 71 is herein referred 
to as a personality port. Each of the personality cards inciude a memory RAM 29c to store data which is to 
be transmitted to/received from bus 71 . Also, each of the personality cards includes a processor device 29b 
which controls memory operations on RAM 29c and controls the interface of the personality card 29 to bus 
40 71. In addition, each personality card 29 includes a Non- Volatile Random Access Memory (NVRAM) device 
29e which is used to store card specific information such as serial number, revision number, manufacturing 
specific information, infornriation related to failures during system operation and diagnostic information. 

Referring now to Figure 2C, a typical network interface (Nl) personality card 30 is shown to include a proc- 
essor device 30c coupled to bus 71. The processor device 30c performs all RAM related operations on RAM 
45 unit 30b, such as read operations, write operations, refresh, related arbitration, and data buffering. The RAM 
unit 30b of the preferred embodiment includes eight 32Kx8 static RAM devices and is used to store information 
received over bus 71 and either a thickwire Ethernet bus 30f (16 bit bus) or a thinwire Ethernet bus 30g (serial 
bus). 

Each network operates under a given protocol which determines the format of data which may be trans- 
50 mitted on the network. In an Ethernet system, data is transmitted in blocks of data, referred to as 'packets'. 
The block of data includes a destination field which indicates the computing system to which the data Is to be 
transmitted. In addition, other informatbn Is included in the packet such as the number of bytes of the data 
field in the packet, a Cyclic Redundancy Code (CRC). and a sequence of bits which signals the Nl card 30 as 
to the start and end bits of a packet 
55 Specialized devices, known to those skilled in the art recognize and respond to the various network pro- 
tocols. The processor device 30c also provides bus 71 access to a Local Area Network Device for Ethernet 
(LANCE) 30d In the preferred embodiment to respond to an Ethernet network. However, other specialized de- 
vices designed for a different network protocol could be used In a different network configuration. 
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Among the functions of the UNCE device 30d is to recognize and respond to network messages destined 
for the computer system which It supports. An Ethernet Address ROM 30a. coupled to the processor 30c Is 
♦ ^^^"^^ l'^'^^^ personality card. Typically, during power up. the network address Is 
read out of the Ethernet Address Rom 30a and stored in the LANCE device 30d. However, in the fault tolerant 
environment, It is not advantageous to use this address, because many Nl cards with different addresses in 
tne Ethernet Roms could be configured in one computing system, and would thereby respond to different net- 
work transactions and fall outof synchronizatton. Subsequently, a Non-Volatile RAM 20 is included on control 
pan^^g to allow for storage of a unique system identif icatton number as win be discussed in conjunction with 

I Awr?*,nJ?r"°'**^2?"'^ "'^'^"^ embodiment, the system Identification number to which the 

LANCE 30d (Figure 20) responds is derived from NVRAM 20 (Figure 1B) mounted on the control panel 19 
(Figure 1B). Because the NVRAM 20 is socketted on the control panel 19. it Is physically separate from the 
remaining computing system, and is not affected by the swapping out of various system components. Although 
in this embodiment the NVRAM 20 is stored on the control panel 19, the system Identification number couW 

1^ ?TJ ^" ^ °" ^^''^^^ component (for example the system console unit) which is 

less hkely to be removed from the feult tolerant system 10. i '» 

iH.nmiT*'°"^'^.'^*2°x,o.')^f """^ ^2b of Figure 2D. during initialization, the system 

.dent f, cation number (SYSID) is read out of the NVRAM 20 on control panel 1 9 and stored in sys^m memory 

are^S: to tht'Z^M' '"'.f^ ^^^'^ ^ ' ^^'^ning I/O device addresseTwhS 

are wntten to the NVRAMs on the appropriate personality cards. For example, the NVRAM 30e (Figure 20) 
on Nl personality card 30 includes an I/O devbe address derived from the original SYSID stored In NVRAM 
zu on the control panel 19. 

„v.,T'.!o k'° o!!'^ ^'"'r^sses have been assigned, data from the NVRAM 20 on control panel 19 Is read 
over the PC bus 25d by the CPLAMemory unit 15. and written to the LANCE device 30d via bus 71 and the 

JIT' ^f: *° <««rived from the memory 

chip information on the enclosure, rather than information stored on the l\0 module or personality card Be- 
cause the system identlfk»tion number is stored in an NVRAM 20 mounted on the control panel 19. the system 
identifi^tran number of the dual-zone system will be unaffected by the replacement of any CPU/Memory 
Unfts. I/O units, or Network Interface personality cards. Detells concerning the accessing of the NVRAMs 20 
30 by the zones 12/12- will be discussed below. w me nviwms zo 

Referring now to Fig. 3A, a block diagram of the cache 46 for the single fall stop computer system 13 Is 
showntoindu^deacachecontroller47aforp^ 

; 47a/47a' are fed via appropriate control signals such as Read. Write and IN- 

n ^^^^ respectively from each of the primary and phantom rails41/4r. The 

MChe controlter further includes cache tag store 47b/47b'. ECC stores 47c/47c- and a single date store 47d 
The cache teg circuits 47b/47b' are fed by portfons of the address Bus 43a/43a' here the bits <29-5> The 
^che teg stores 47b/47b' are used to Wentif y the corresponding locations for which stored ECC bits in the 
ECC stores 47C/47C and the date bits stored in date store 47d are stored 

roll Ia^^'I^.T^'^-^^''^^ ^ * corresponding date store for the date from the phantom 

rail 41 and that there is not a corresponding connection of the date Bus from the phantom reil Into the cache 

"T""'' '^^"^ ope^'tons the date bus from the primary reil couples the date onto 
bus 43b to input into the phantom rail processor 42". 

Caclie 46 is typically used as a relatively fast access portion of storage for the central processor 42/42" 
The single date store 47d occaskined by the use of an abbreviated date Bus in the phantom rail 41' as men- 
twned above and as will be further discussed below, reduces the date store requirements for the cache store 

H. i, ^T!^-^"^? °" ^'9"^'^ sf'"*") generated by the cache con- 

roter 47a/47a to permit date on the address lines 43a/43a'. ECC lines 43c/43c'. and date on date lines 43b 
to be Stored respectively in the cache teg stores 47b/47b' ECC store AldAld and the date store 47d 

Referring now to Figure 3B. the memory subsystem includes a primary rail memory controller 52, a main 
^rJ^'Z ^?^^'^. """'""'^r 52-. Referring now to the primary rail, the bus interface 

cn-ojrt 54a of the pnmary rail 41 (memory controller 54) controls memory access signals (such as RAS, CAS 

£L rA??;^ -^^ °" ^"^^ ^^"^ ^S'^ ""^y "P "fts comprised of both 

oata and ECC to provide a communicatfon link between the 128 bit date bus 43b and the 28 bit ECC bus 43c 
of the pnmary CPU 42 and the 256 bit date bus 55b and the 56 bit ECC bus 55c of the memory 50 
^ h..«r Tc'I^c f li^Tr'""'^'"^ ^"^^ ^^'^^ "^^'^ 52/52' furnish control and select signals to the 
memory address select bite for memory operattons from the CPU 42/42' to address and control asic 54a. while 
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memory address bits for DMA operations are provided by interface and control circuit 60. The main sequencer 
PALS also provide select lines to the 4 buffering ASICS to control which longword of data from memory Is driven 
on bus 43b. 

The phantom rail 41' Includes bus Interface circuit 54a' which provides menrwry control signals identical 
5 to bus interface circuit 54a of the primary rail. The ECC/data storage device of the phantom rail Includes only 
one buffering ASIC 54b' to store only ECC data bits from the ECC bus 43c'. Because the data bus is not pro- 
vided in the phantom rail, 3 fewer buffering ASICS and less space for conductors which would have been re- 
quired to replicate data bus 43b are needed. Similar to the primary rail, the main sequencer PALs 52' provide 
memory address select bits to address and control asic 54a' and buffering asic 54b' for memory operations 
10 from the CPU 42/42', while memory address bits for DMA operations are provided by Interface and control 
circuit 60/60'. 

Main memory 50 includes four memory motherboards 82, 82a. 82b, and 82c. While all four memory moth- 
erboards are Identical, memory motherboard 82 has been shown in detail and all references to niemory moth- 
erboard 82 should be understood to be applicable to all four memory units. 

15 Each memory unit 82 is capable of supporting 2,4,6, or 8 memory SIMMs (Single Inline Memory Modules). 
Each SIMM contains either 10 or 20 DRAMs (Dynamic Random Access Memory). Each memory motherboard 
82 holds two longwords of data (64 bits) out of every hexvford (256 bits). A SIMM with DRAM devices on one 
side has one bank of memory, a SIMM with DRAM devices on both sides has two banks of memory. The menv 
ory motherboard 82 shown in Fig. 3 includes 8 SIMMs each having DRAM devices on both sides of the unit 

20 Depending on whether 256Kx4, 1 Mx4 or 4Mx4 DRAMs are used in each bank, the maximum bank size is 8MB, 
32MB, or 128 MB. 

The memory motherboard 82 also includes ECC checking and generation logic 86 for ECC checking of 
two longwords and comparison check logic 88 for comparing ECC, address, and control information from each 
of the memory controllers 54a/54a'. 

25 The ECC check and generation logic 86 generates 14 ECC bits for the two longwords of data which It re- 
ceives from the memory bus 55b. The generated ECC bits are compared to the 14 ECC bits received on the 
primary rail ECC bus 55c. Single bit errors and multiple bit errors during memory reads and memory writes 
are indicated by assertions of signals SINGLE BIT ERROR and DOUBLE BIT ERROR, and the error is logged 
In error register 17 on the CPU unit The primary rail 41 ECC error code bits on ECC bus 55c are then stored, 

30 along with the data bits on memory bus 55b, Into the memory SIMMs. 

In addition to the ECC check of the data discussed above, the ECC bits on primary rail ECC bus 55c and 
the ECC bits on phantom rail ECC bus 55c* are checked for inconsistencies by the checker logic 88. The check- 
er logic 88 addlttonally checks the address bits on address bus 55a of the primary rail with the address bits 
of the phantom rail bus 55a', and the memory control signals from the address and control logic 54a on bus 

35 55d with the memory control signals from address and control logic 54a' on bus 55d'. If an inconsistency is 
detected, signal ECC COMPARE ERROR , or ADDRESS AND CONTROL COM P/VRE ERROR is asserted, and 
an error is logged in the error register 17 on the CPU unit 15,15' (Fig. IB). One error bit associated with each 
of the 4 types of errors (single bit error, double bit error, address and control compare error and ECC compare 
error) is stored for each memory motherboard (82, 82a, 82b, and 82c). 

40 When data is written to memory, the ECC check and generate logic 86 may detect a single bit error on 
the write data. Similarly, when data is read from the memory, the ECC check and generation logic 86 provides 
14 bits of ECC bits for the two longwords of read data, and compares these ECC bits against the stored ECC 
bits. Both the primary rail 41 and phantom rail 41' receive the stored ECC bits on bus 55c along with the data 
bits on bus 55b. The read data and ECC are temporarily stored in the ECC/data storage device 54b on the 

45 primary rail 41, and in the ECC storage device 54b' on the phantom rail 41*. 

The data bits and ECC bits are transmitted from the memory controller 54 in the primary rail 41 by ECC 
bus 43c and data bus 43b and the data is checked and/or corrected in the CPU 42. The phantom rail 41* re- 
ceives only the ECC bits on ECC bus 43c' from memory controller 55', however, it receives the correct data 
bits from data bus 43b, thus enabling the two processors to maintain synchronous and accurate operation. 

50 The CPU 42' of the phantom rail 41' also checks and corrects the received data bits. 

If there is corruption on the data path during the read operation, both CPU 42 and CPU 42' will detect an 
error. If an error is detected by either CPU at this point which was not detected by the error checkers in memory 
logic during the memory read, then the fault may be isolated to either memory controller and their correspond- 
ing data busses. 

S5 Thus, the error checking mechanisms operate collectively to notify the system of a fault within one of the 
zones In order to ensure that a faulty zone is immediately removed from the system. Consequently, fault tol- 
erance is maintained without a replication of a cache data storage device or a data bus on the phantom rail 
41'. Hardware expense is minimized by checking the data bus for CPU 42' on the phantom rail. By checking 
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memory read and write data by the memory device 50 fault isolation to a module level is ensured. 
Error Code Detection/Correction 

' ECc'itfiTL^^^ '".oS' ^^'"P'^^^ invention uses 7 bits of 

fo^le^.fJ^^^^ " • "^'^^ ^'^^ ^•^'^ ^^^'^ an X In their column 

rrjr^vl; • f ^ ^ exclus,vely.ored together, while the data bits used to calculate ECC bits g2 and 
03 are exdusively-nored together. For example, data bits 31. 29, 28, 26. 21.1^^ 14 11 9 a Ts 4 

fo and 0 are Exclusive-Ored together to provide ECC bit cO ' ' ' ' ' ^ 
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As shown in Table 2, (wherein only 16 bit of the ECC table shown in Table 1B are used), to calculate the 
check bits for 16 bits of data 0011001100110011 (wherein the bits are ordered from bit 15 - 0) each bit value 
Is written in Table 2 where a corresponding X appeared in the Table 1. Therefore, to provide ECC bit cO for 
50 data 0011001100110011, data bits 0, 4, 6, 7, 8, 9, 11 and 14 are exclusively-ored together, as shown below: 
(1XOR1)=0 

XOR 0) =0 

XOR 0) =0 

X0R1)=1 

S5 XOR1)=0 

XOR 0) =0 

XOR 0==>0 

Similarly, the values for C1-C6 may be provided, giving a correct ECC code for data 0011001100110011 
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equal to 0110010, as shown in Table 2. 

As described previously, before data from data bus 55b is written to the memory motherboards 80, 80a. 
80b and 80c, the ECC is generated from the data and checked against the ECC which is sent to the mother- 
board over the ECC bus 55c. If, for example, there was a fault in data bit 10, making the data which was sup- 
5 posed to be 1100110011001100 equal to 1100110011101100, then the generated ECC (calculated in the same 
manner as shown in Table 2) would be equal to 0000000. AXOR comparison of the generated ECC with the 
received ECC yields an error syndrome. 



TABLES 
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The Error syndrome of this example is equal to 0110010. To determine which bit of the data is in error, 
and thus to correct the bit, the error syndrome is compared to the check bit strings (c0-c7) for each data bit 
20 as shown In Table 1 . The comparison Is made to detect which bit location has a pattern equal to the error syn- 
drome. By way of example Illustrating this point, replacing all of the X's of the ECC Table 1 with 'I's, and all 
of the blank spaces of Table 1 with 'O's, it can be seen that the ECC pattern for bit 10 or the data is 011010, 
which is the same pattern as that for the error syndrome. Thus, bit 10 Is correctly identified as the bit in error. 
The ECC check and generate logic 86 can also determine If the error bit was not In the data but In the 
25 received ECC as follows. For example, if the received ECC on ECC bus 55c was equal to 0111010, and the 
generated check by the check and generate logic 160 is 0110010, the Error Syndrome is then 0001000, 
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35 A 1 In the check bit string location c3 thus indicates that check bit c3 was the bit in error. 

Thus, through the use of a combination of minimal hardware replication and frequent error checking and 
correctbn mechanisms, a CPUVMenriory unit capable of isolating faults provkjes a high degree of fault tolerance 
by enabling the detection and correctton of faults within a system and facilitating the return of the system to 
full redundancy. 

40 

Booting the Dual Zone System 

Referring now to Figure 4 bus controller 22 provides an internee between the power supply 4 (Figure 1 A) 
and the byte wide bus 78a (Fig. 2B) to enable the CPU 42 (Figure 2A) to control and monitor certain charac- 
45 teristics of power supply 4 (Figure 1 A). For example, an over temperature condition or variations In input line 
current can be monitored and the power supply can be powered down by software for an orderly shutdown. 

Bus controller 23 provides an interface between the NVRAMs on each personality card (for example 
NVRAM 30e on the Network Interface card 30, Fig. 2C) and the byte wide bus 78a, thus allowing reads and 
writes of the various Information such as serial number, revision number, etc. over bus 78a. Bus controller 21 
50 provides an interface between NVRAM 24 on the CPU/Memory unit 15, the NVRAM 20 on the control panel 
19 , the NVRAM 24 on the I/O unit 25 and the byte wide bus 78a, thus allowing reads from and writes to the 
respective memory devices over bus 78a. NVRAMs 20/20' on the control panels 1 9/1 9' are disposed in sockets 
(not shown) used to store the Ethernet system Identification number for the entire dual-zone system 10. Both 
NVRAMs 20/20' include the same Ethernet system identification number. Each dual-zone system is furnished 
S5 a block of 32 consecutive Ethernet system Identification numbers. The Ethernet system identification number 
which Is stored In NVRAMs 20/20' is the base address of the block of 32 consecutive addresses. 

During the power up sequence, both zones 12/12' operate individually to initialize the clocking system, per- 
form diagnostic testing, load system software, and provide the CPUs 42/42' and memory units 50/50' an Ini- 
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tialized state. During initiaiization. each zone reads a system identification number out of NVRAM 20/20'. and 
loads it into system memory space within memory 50/50' (Figure 2A). 

During the power-up sequence, each zone assigns I/O device addresses to all the external devices at- 
tached to the I/O unit personality ports. Each I/O device address Is derived by adding the port number of the 

5 personality port in which the I/O device resides to the system identification number which was copied from 
NVRAM 20 into system memory. For example, given a system identification number of 100, the personality 
card in personality port 0 will have the I/O device address of 100. the card In port 1 has the I/O device address 
of 101, and the card in port 7 has the I/O device address of 107. If a second I/O unit 18 is coupled to Ml busses 
25a and 25b, the I/O device addresses of the 8 personality cards coupled to bus 26c would range between 

10 108 for personality port 0 of bus 26c, to 115 for personality port 7 of bus 26c. The most significant bit of the 
I/O device address is then inverted for addresses to the other zone. 

Referring again to Figure 20, In addition to the t/0 device address stored in NVRAM 30e, a second system 
identification number is stored in an Ethernet ROM 30a on the Nl unit 30. In single processor or single zone 
computing system, this second system identification number may be used as the Ethernet address. In a single 

IS zone configuration, the system Identification number may be copied from system ROM 30a and stored in the 
l^CE device 30d. Thus, the Ethernet address stored in Ethernet ROM 30a is used as the network address 
for the single zone system. However, in the single fail stop computer system 13, the Ethernet address which 
is written to the LANCE device 30d is derived from the system identification number stored in NVRAM 20 on 
control panel 19. As discussed previously, this is achieved by the CPU/Memory device reading the system iden- 

20 tification number from the NVRAM 20 on control panel 19 and writing, and writing it in the LANCE chip 30d 
via bus 71. As discussed previously, the address stored in the LANCE device Is considered to be the network 
address of the dual-zone system 10, and it Is the address to which the LANCE device 30d responds to transmit 
and receive packets over the network. 

Although the two zones are not Initially in synchronization, communication between the two zones may 

25 still be achieved over the serial crosslink communication channel 25f. At the end of the diagnostic testing, when 
a zone has been determined to operable, it uses the serial crosslink communication channel 25g to check the 
other zone to determine if that zone has also finished its diagnostics. The first zone to complete diagnostics 
designates itself as a master zone by setting a master flag in the crosslink communication unit 65. The second 
zone to complete diagnosttes designates itself as a slave zone by setting a slave flag in the crosslink commu- 

30 nicatlon unit 65. If both zones should happen to complete diagnostics at the same time, one zone, which is 
Initially designated as the "A" zone would always default to be the master zone. 

After the master zone and the stave zone have been determined, the diagnostic results of each zone are 
compared to detect any inequalities. One of the Items which diagnostics checks for consistency is the system 
identification number which was copied out of NVRAM 20/20' and written to memory unit 50/50'. If there is an 

35 inequality with the results of one zone, there may be a fault in one of the dual-zone systems. The zone without 
the fault commences operation, while the faulty zone is removed from the system. The faulty zone Is repaired 
or replaced. After two functional zones are provided into the dual zone system, the new or repaired zone is 
synchronized with the operating zone to restore complete fault tolerance to the entire system. 

Referring now to Figure 5A, if there are no inequalities between the diagnostic results, at steps 101/1 Ola 

40 and 102/102a each zone determines whether it is the master zone or the slave zone. Initialization of the dual- 
zone system 10 begins by the two zones, convnunicating over serial bus 25e (Figure 1 B) to agree to establish 
re-synchronization mode. First, the clocks are brought into synchronization by locking the slave clock to the 
master dock. This process is described in more detail later in the specification. Once the clocks in the two 
zones are brought into lock step, the two zones are operating synchronously. 

45 During the initial phases of re-synchronization, the slave zone disables Its cache 46 (Figure 3a) at step 

1 03a, and at step 104a configures its main nfiemory according to data received from the master zone over bus 
25g (Figure 1 B). At step 1 05a, all of the error registers are cleared, and error reporting is enabled. At that point, 
at step 106a. the stave zone sets itself as the 'resynch stave' and proceeds to step 110a wherein the slave 
zone executes a code loop which precludes the CPU/Memory Unit 42 of the slave zone 12' from accessing 

50 Its memory 50 or its coupled I/O Unit 25. The memory 50 in the slave zone is here dedicated to write operations 
directed from the master zone. 

During the initial phase of resynchronization, at step 103 the master zone transitions the cache 46 (Figure 
3a) to write-through mode. In write-through mode, each time data is written to the cache, the data is also written 
to main memory. As mentioned previously, in step 104 the masterzone then configures the slave zone memory 

55 structure to be consistent with its own memory structure using bus 25g. At step 106 the master zone checks 
to see whether the slave zone is functioning as 'resynch slave'. When the master zone detects 'resynch 
slave' from the slave zone, at step 106b the master zone sets itself to be the 'resynch master'. At step 107, 
once both zones have entered either 'resynch master' or 'resynch stave', the entire contents of main memory 
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50 Of the master zone are copied to main memory 50 of the slave zone. This is accomplished by performing 
memory writes which are transferred to the otherzone via the re-synchronization bus 15e 

After all of memory has been copied, at step 1 08. a RESTART PC is loaded, and the CPU state from the 
rnaster zone is placed mto memory in both the slave zone and the master zone. At step 109. the master zone 
then initiates a reset of the dual-zone system, and at step 110 enters a wait state 

rD. 1°??^ ■ ^® reset process, the contents of the caches 46 are invalidated, and the caches are enabled. The 

foJlc ?,Mo^ ■ ^"^ ^"'"^^ ^y^"^ '^'^y opefat'o"- At this point both 

zones 12/12 are operating on an identical instruction stream simultaneously 

in 'inrJlw ^^1, ^""^^ '° «Peration, or remain 

; H^ii^'h H '^^'^ °^ "'^y*- ^^'^ '^o^^ '<«e phase lock. This 

Iri?^?^ c 'H communication devices 65/65' (Figure 2b) and handled as a system 

TnT/n L K ■ ^ ' ^"^"^^ ^"^^ ^ ^y^*^'" P^'fo^-^s an operation which aa^sses 

an I/O device checks are performed by hardware to make sure both zones are doing the same I/O access 
However. ,f they are not both performing the same access, lock step is lost. This condition is detected by one 
of the crosslink communications device 65 or 65'. as soon as one zone attempts to perform an I/O operation 

I.k ilc^n'irt ^ T**" ^""^ ^"^ °^ the system configuration. In 

m« instance, because it is unsure exactly which zone is in error, the zone which is initially designated as zone 
A continues to operate while the other zone is removed from the dual-zone system A zone may also Z 

20 Z^TlT. ^ ' ''"""^ "^-^ -"^^^y 5° (Figure 2a). or a misctiipare of 

20 data along the primary and phantom rail within a zone. 

«f ^[""^T ^' ^''^ 3 ^3ult and signals the 'surviving' zone 

ofrtsfaulcond,tK>n. At steps 113and 114. when one zonedeteotsafaulUhe 

as a masterzone and continues operation on the instruction stream Independently. The type of failure which 
caused the fault is determined thrxxigh examinatton of the faulty zone 

^Jl^2^ fSTl^?^!!^ zone or the control panel needs replacement in a dual-processing 
IS' f ^ '^'^'^ ^° ^'''^"'^ '^"-^ Position on control panel 19 

When 1 inl^^^h ""^ ^ " ''"'^'^ P"^*" ^ P°«'«°" *e new zone. 

When a zone wtth the new zone or control panel is replaced in the dual-zone system, it powers up at step 121 

' system Wentification number out of the NVRAM 20 on control panel 

19. initializing Its I/O device addresses, and designating Itself as a slave device 

1 9 S thTduS """^ r ' ■ ri"™^ '^^"^ Becauie the contrel 

J^Zl^ ! T!^ ™* "''^'^ """"9 °^ CPUVMemory unit, the technique 
o assigning the systemidentifteation number to thezo 

er-up of the new zone then proceeds as discussed above. » » j~w 

o oi '^Lff !!^« V* '^^^'"''''^ « « fa"'t with the socketed NVRAM 20 on the control panel 19 

a slightly different process Is used to bring the second zone back into the system. Only the NVRAMs which 
were ongina liy placed on the control panel include system identification aSresses. If. at step 118 a n^J 
NVRAM IS placed in the control panel, its contents will be blank. Thus, after the zone finishes the power up 
sequence descnbed above, the system identification number will be 0000. 

Once the faulty zone has been repaired and finished powering up In step 121. at step 122 the repaired 
zone request r«synch«,nization from the 'survwing' zone. The request for resynchronization is made over bus 

and?h?S'srnTl^'T^'"'^^^*^'^''"°"^*°'^'y"^^^^^ 

JSiiSL I! « 1 " diagnostic results and the SYSID do match, at step 

1 02/102a the master zone allows re-synchronization to begin as discussed with reference to figure 5A. 

if the diagnostic results do not mateh. the master zone does not allow the zone to re-synchronize and 
returns the slave zone to FAULT state at step 112. If the SYSID does not match, at step 125 the maSr zone 

h«nM.? ?H "i " •^^^"P ^'"'e zone, and rather 

than allow the slave zone to re-synchronize, at step 112 the master zone returns the slave zone to a FAULT 

with^h?sS^fi^r,°h!'l?AM 

with the SYSID from the NVRAM on the master control panel, then at step 102.102a. the master zone starts 
the re-synchronization process discussed with reference to Figure 5A 

ERi iw ' ^ ^^^*f^ Wentification number Is maintained within a dual-zone system regardless of the attached 
FRU 8. Aprocess fbrreplacing the NVRAM including the system identification number precludes the intreduc 
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tion of more than one dual-zone system having the same system Identification number. 

Referring now to Fig. 6A, the Bus interface and control circuit 60 is shown to Include a resynchronlzatlon 
control Interface 135 which is coupled to a dedicated resynchronlzatlon Bus 15e and Is used to permit one of 
said zones 12/12' of the fault tolerant computer system 10 which has lost lock step or for which a fault was 
5 detected to be resynchronized with the other one of said zones. The resynchronization interface 1 35 produces 
output signals to a data path circuit 130 to permit data along resynchronlzatlon Bus 15e to be fed through the 
data path circuits 130 Into the primary and secondary rails of the respective zone 12/12' which is being re- 
synchronized. 

The Bus interface and control circuit 60 further includes a data path circuit 1 30 which interfaces the circuit 

10 60 to the Internal Interface Bus 56 and thus which is fed by the address, data, ECC bits, and control signals 
from Bus 56. Correspondingly, the ECC bits from Bus 56' are fed along ECC Bus lines 56b' to circuit 60 and 
the corresponding ECC bits from Bus 56 are fed to the corresponding Bus interface and control circuits 60' In 
the phantom rail 41'. The ECC bits from each rail are compared in the Bus interface and control circuits 
60/60' to ensure that CPUs 42 and 42' (Fig. 1) are operating in lock step. 

15 The Bus Interface and control circuit 60 further Includes a DMA / I/O decoder circuit 134 which is fed ad- 
dress signals via bus 56a to decode the circuit address. The DMA and I/O interface decoder provide control 
signals to the data path 60 to select the route of data through the data path to the remaining portions of the 
Bus interface and control circuit 60. One of the paths through the Bus interface and control circuits 60 Is via 
a Bus 130a which couples address, data, and control Information as necessary to a DMA input / output inter- 

20 face circuit 146. The DMA control Is provided by a DMA control circuit 134b which is fed signals from control 
bus 56d. The DMA control circuit 134b is a synchronous state controller, the details of which will be described 
below. Also included in DMA control circuit 134b is the I/O controller which controls input/output operations 
between the I/O interface and control circuit 60 and the I/O bus 15c. The DMA input / output interface circuit 
146 provides a DMA transfer between the main memory 50 (Fig. 1) and the Bus Interface and control circuit 

25 60. The data path circuit 130 also provides Bus connections to a register stack 129, a resynchronization inter- 
face 135 which is coupled to a dedicated resynchronization Bus 15e permitting one of the aforementioned conv 
puter zones 12/12' of system 20 which had lost lock step operation with the other one of said zones through 
either a fault or replacement to be resynchronized with the properly operating one of said zones 12/12*. The 
data path 130 further provides Bus 130a to a ROM fnterfece 145 which Is coupled to a dedicated ROM Bus 

30 15b. 

The Bus Interface and control circuit 60 further Includes X dock error circuit 138 which Is used to determine 
system response/action when errors are detected. The X dock error circuit is used to handle all error conditions 
that can be handled under X dock (CPU dock) side of the system. For those errors which need immediate 
attention and which arise under the P dock side (I/O side) of the system, a P dock error circuit 148 is provided. 
35 For example, a X dock error could be an attempt to access a non-existent I/O address, whereas a P dock 
error condition would be an error related to Inter-rail checking operations being performed. Exact routines 
would be provided to accomplish these tasks as would be apparent to one of ordinary skill in the art 

The P error circuit 143 is provided for preprocessing error reports on the P side of the system. 

The protocol synch ronizatk>n circuit 142 Is used to transfer In a synchronous manner signals between P 
40 dock and X dock boundaries. 

In addition, the Bus interface and control circuit 60 further indudes a P dock and X dock alignment drcuit 
140,144 which is part of the dock interface circuit 92 (Fig. 2A) as will be further described in conjunction with 
Fig. 10. Circuits 140, 144 generate a plurality of phases for each of the respective clock signals and provides 
status type signals to corresponding circuits in the other rail to Insure that each rail in each zone uses the 
45 same respective clock phase. Suffice It here to say that the Bus interface and control circuit 60 in addltk)n to 
providing the aforement toned functions also provides a circuit to interface a dock which would operate an I/O 
type of device to a dock which operates the CPUs 42. 42' in the respective primary rail 41 and phantom rail 
41' computers. 

Referring now to Fig 6B. the Bus interface and control circuit 60 is shown to indude a DMA/ I/O interface 
so circuit 146 which in conjunction with the DMA control circuit 1 34b (Fig.6A). controls DMA transfers in primary 
rails of the two zones A, B. 

Data flow through the DMA / I/O Interface circuit 146 is as follows: a DMA subtransfer buffer circuit 136 
is used to store data during a direct memory access transfer between I/O and main memory as will be further 
described in conjunction with Fig. SC. From the DMA subtransfer buffer 136. data from I/O data bus 15c on 
55 line 1 86 or data from data path 1 30 via bus 1 30a will be selectively coupled through a MUX 1 84 to a DMA read 
modify write buffer 1 82 as will be further described with conjunction Fig. 60. The output of the DMA read mod- 
ify write buffer 182 is fed to the input of a multiplexer 190a to feed output data to error check and correctkm 
circuit 131 as will be described in conjunction with Fig. 6C. 
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/n.f 1® ^ ^ ^"^^ 180 Which is loaded from a register 

(not shown) which is coupled to bus 56. The DMA Interface further includes an address pipeline stagecom- 
pnsed of nine registers 180a-180i, five clock stages or registers operating at an X1 dock phase and four X3 
clock stages or registers each disposed intermediate pairs of the X1 clock phase register operating at an X3 
clock phase. The output of the last one of the registers 1801 in the stage 180 is coupled to the Input of a MUX 
1 75 which feeds an address out to register 176 and to the Bus 56 via a Bus interface buffer 176a or alternatively 
feeds the address through MUX 190b and 190a through the error check and correction circuit as data which 
signifies the address of a word which was detected as being incorrect during a DMA read of the memory as 
will De described r)elow. 

The address interface 132 further includes registers 170 and 172 operating on respect Krely X3 docl< and 
^J^a"1 ^ generator circuit 173, as will be described with conjunction 6E, which Is used to check 
party 0^ data on the address Buses 56a/56a'. Each of the primary and phantom rails supplies a parity bit rep- 
resentatn^e of the panty of the I/O address to the parity generator circuit 173 or 173' in the other rail. The output 

3? ir»K ^.l^ ^^"l?*" P™^'**^ P«"'y circuit in the other rail to ascer- 

temmg whether the parity bits are equal, and thus whether the address lines 56a/56a' have the same infor- 

„«Ht''^T®^^* °^ ^'^^^ ^^"^^^^ ' ^^^^^ ""^"'^ ^ '® « ^^^^^ ^ ^^ich stores the generated 

party of the I/O address. The output of register 174 is coupled to an address register 198a and an I/O address 
register in the I/O interface 189b. 

^,2T^^^°JT^..«'T^ " P'"^'"^ '^'^^"^ ^^^'^^ ^5cor bus 56 to fill DMA subtransfer 
^^^.^1 riZ^® ^'"^'^ "^^^ ^^'^ °" P '^"^ ^) or the X clock (CPU dock) 

Since the DMA'nterface can write to main memory on byte boundaries, that is it uses long words as the smallest 
address entrty. but the memory controller 152 uses longwords as the smallest addressable location the Bus 

Tl"^ ^ P"""**®* ^ ""^'^y cycl* «" appropriate byte mask to permit in- 
dividual bytes to be nmdif led in memory, -r K , «.uj,«rmiiin- 

During a DMA write operation, the read modify write register 182 receives the modified long word when 
a byte mask with zeroes is provided to the DMA control drcuit 134b. The byte mask register wfth any zeros 
T^J!^ ^k"^"!! ^ °^ '^'^y and penbrms a read of memory at the current address 

location in the address register 1801. When the data Is returned to the Bus 56b, the data is written into the 
read modify write register (182) with the byte mask bits inverted. The DMA write transfer Is then resumed and 
tne write is completed to main memory. 

When the DMA/ I/O interface circuit 146 has completed loading the subtransfer buffer 136 with data it 
will assert signal DMA WRITE to the DMAcontrol circuit 134b. The DMA/ I/O Interface dnnilt 146 also asserts 
ttie address of the first long word In the address register 180. The DMAcontrol circuit 134b will move the ad- 
dress up through the pipe delay network (register 1 80a - 1801) and assert the address on the address Bus 56a 
to interface circuit 54. It will also assert the longword offset in a hexwortl on the data Bus 56b to each of the 
interface crcu tts 54b - 54e. The signal LOAD DMA ADDRESS is asserted to the drcuit 54a and signal RAM 

Mb-54e. Date will movethreugh date path 133outoftheDMAbuffer136alongwor^ 

the read modify wrte buffer 182. The read modify write buffer 182 acts as a pipeline delay stage. Frem^e 

J^-I J^' ?: V IT ° "^^ '"'"'^'^^ *e bytes of the longword are 

vahd then data transfere out of the subtransfer buffer are halted. The pipeline above the read modify write 
l«3«r« '"""Pt'*! f the contorts thereof are written to main memory and a read modify write operation 
IS performed as descnbed above. Otherwise if the long word is complete the DMAcontrol circuit 134b cortin- 
ues moving longwords up through the MUX 190a and dicuit 131 and out orto the Bus 56. One dock cyde 
R AM wp'rfc 'T*** °" ^' DMA"'"*'^ circuit 1 34b will assert a signal RAM Wr7te 

RAM WRITE wil remain asserted urtil the complete data transfer from the subtransfer buffer 136 has been 
T^Tl ^ de-asserted one clod< cyde before the last longword is inserted 

!o ! ^^^.2"^ '^'"^ ""^^ '>°""*'ry reached, the control logic will 

oMccT ^ REQUEST to the main sequencer PALs 52. Within three to six dod< cydes of the signal DMA RE- 

2S i« ""f w ^l^^Z "^"^'y °' ^'^o'^ing to the particular 

Sl?f J^t^^A "^^^ '^'^^ ^^'^ ^"-"P'^t^ sequencer 

di.^,lrt th HMA ^lin^^x '"^"'^"'^ '"^ "^'""^ Tl^^ DMAcontrol drcuit 134b will 

I » K ^^^^^l ""^ ^ """"'^'^ ^^"^^ « DMA buffer (not shown )in circuits 

541^540. If It has a complete hexword it will leave DMA REQUEST asserted. One dock cyde after the DMA 

'* ^"^^ 'NCREMENT READ POINTER and INCREMENT DMA 
ADDRESS for one dock cyde. When the last k>ng word is removed from the subtransfer buffer 1 36. the DMA 
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control circuit 134b can allow the DMA/ I/O interface circuit 146 to start another subtransfer by asserting DMA 
DONE. 

During a DMA read fronr) memory 50. the DMA/ 1/0 interface 146 asserts DMA READ and places the DMA 
memory address In the DMA address register 180. When the DMA control circuit 134b detects DMA READ it 

5 will move the address up through the rail time delay pipe 180a-180l and insert the address on the address 
Bus 56a. The DMA control circuit 134b will then also insert the long word offset in a hexword on the Data Bus 
55b to interface circuits 54b - 54e. While the address and long word offset are asserted, the DMA control circuit 
134b will assert LOAD DMA ADDRESS to the Interface circuit 54a and signal RAM RESET through the data 
interface circuits 54b - 54e each signal for one dock cycle. Immediately, after de-asserting LOAD DMA AD- 

10 DRESS and RAM RESET the DMA control circuit 134b will halt the placing of data on the Bus 56. During the 
following clock cycle the DMA control circuit 134b will change the control select lines to indicate DMA READ 
to interface 54a and also a signal DMA REQUEST to the main sequencer PALs 52. In response to these signals 
the main sequencer PALs 52 generate a read cycle to fill buffers (not shown) in data interface circuits 54b - 
54e with data within the next three to six clock cydes. The main sequencer PALs 52 will then assert the DMA 

f5 DATA signal, indicating that within two dock cydes the block from memory will be ready to be copied Into the 
DMA subtransfer buffer 136 In Interface and control circuits 60. One dock cycle after DMA data signal has 
been inserted by the main sequencer PALs 52, the DMA control circuit 134b will assert DMA WRITE for one 
dock cyde which will cause the data to get latched into the data interface circuits 54b-54e on the following 
dock cyde. The control logic will also assert INCREMENT DMA ADDRESS during this time for one dock cyde 

20 which causes the DMA address pointer In address Interface circuit 54a to be incremented. After the DMA DATA 
signal has been asserted, the control logic 134b will assert INCREMENT READ POINTER and remove the 
first tongword off the Bus 56b. As long as INCREMENT READ POINTER remains asserted, data will be placed 
on the Bus 56b in sequential order from data circuits 54b-54e. As each longword of data is removed from bus 
56b. the DMA address pointer incremented, and the incremented value is stored in the DMA address register 

25 1 80. then flowing through the DMA address pipeline 1 80a-1 801. 

As data is loaded into the sub-transfer buffer 136 of the interface and control circuit 60 , error correction 
will be performed on each longword passing through cirouit 1 31 . The error correcting check will take one dock 
cyde. The DMA control logic 1 34b will place each long word Into the sub-transfer buffer 1 36. The control logic 
134b will always request at least thirty three longwords of data even if the actual DMA sub-transfer is less 

30 than thirty three longwords. When the DMA subtransfer buffer is full, the DMA control drcult 134b will assert 
DMA DONE to the DMA interface 146. When the DMA/ I/O interface 1 46 detects DMA DONE It can begin taking 
data out of the buffer a long word at a time. If more subtransfers are required, the DMA interface 146 will again 
assert DMA READ and continue to do so until all DMA sub-transfers have been completed. 

Referring now to Fig. 6C, an error correction code check drcult 131 is disposed in the data path circuit 

35 130 and is shown to include bidirectional buffer circuit 200b. 200b' and 200c respectively interfacing the data 
bus 56b and error correction code bits on Bus 56c, 56c* to the error correction circuit 131 and hence the bus 
interface and control circuit 60. The output from the bidirectional buffer 200c is fed to a first register 202a op- 
erating at clock phase X3. Output of register 202a is coupled to a second register 202b, operating at clock phase 
x1 , to store subsequent portions of the data from bus 56b. The output of register 202a is also coupled to an 

40 error correction check circuit 204a. A simi lar arrangement is provided via register 203a and 203b for ECC bits 
from the primary rail 41 provided atthe output of bidirectional buffers 200b. Registers 203a' and 203b' provide 
a similar arrangement for the error correctbn code check bits from the phantom rail 41' via the buffer 200b'. 
The output of registers 203b, 203b' are fed to a bank of "exdusive-OR" gates 208 to provide an indication of 
whether the primary rail and phantom rail check bits are equal and thus whether the prin^ary rail and phantom 

4S rail are operating on the same data and thus are presumed to be in lock step or synchronized operation. 

The check bits and data are fed to the error correcting code check circuit 204a and the output is an error 
syndrome which is stored in register 204b. The syndrome stored in register 204b is a string of data bits which 
indirectly identifies the location of and number of errors, if any. in data fed to a circuit 204a. If the syndrome 
is equal to zero (i.e. if the string of bits here are all zero) this Indicates that no error occurred In the data. If 

50 the string of bits is not equal to zero than one or more errors occurred In the data. Here error code checking 
and error code generation Is provided as mentioned above in conjunction with Tables 1-3. The syndrome Is 
used as an address for a look up table which ascertains which bit or bits of the data stored in register 202a 
are in error. Here the code used is capable of correcting one bit in error. If the syndrome can be decoded from 
the look up table, then the data Is capable of being corrected and Is fed to a corrector drcult 204c. However, 

ss if two or more errors are detected then the error can not be corrected and a signal Is provided on line 204 
from corrector 204c to Xerror circuit (1 38 Fig. 6A) to indicate to the CPU 42 that a mufti bit error occurred . As 
a result the CPU takes an appropriate action such as removing itself from processing. 

Thus, if the syndrome used as an address to the look up table (not shown) produces a decoded value which 
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can be used to correct the data, this syndrome is then fed to the corrector circuit 204c. In addition the data 

rScTo^ri'? ^°T:^h' ^" '''^'^ ^"^ "-"^ passed to tJeirr^r 

circuit 204c. Both ttie syndrome and the data are used in the correcter circuit 204c to provide corrected data 

Zd/n H 'fr/T? '^'^ «"''«e<'"e""y to register 207b. Thus, the ouS,ut f r^^^str 20^ 
S »H ?1 J ""^ 207a is provided on bus 130a to the remaining ^rtlons 

of the interface and control circuit 60 as shown. •"■"aming portions 

^nrvfii'*"^; ^"'"^^'^ ^ """^^ 208c to be fed to an output buffer of transceivers 

Thf 7'°'''",' 11° ^"""^ ''''''' P^"*'"'* correcting codes in accordant witS 

S)2 and th! to;^'«te^208b to provide error correcting codes to the primary lail 41. via b^fS 

?S1 ^l, ?° '° '^^'^ *° P""'"^ "'^ '^l' via buffer 200b-. 

These codes are fed out along the primary ECC and phantom ECC Bus 56c/56c' as shown 

aen^^iTcT™ ^^'^^^ T "^!*"^ """"^"^ f^"" computer systems and computers In 

Tdom ^vin. T"'^ ^ -""""^y- "^^^^ "'^^ on « ^««>"ent baste due to 

random events such as alpha particle disruptions of memory cells. These errors are called soft errors sinS 

theie IS noelecr,<^ failure of the cell. The error needs to be removed orscrubbed to prevent the^^^^^ 

dL'^^to2o^^^^^^^^^^^ r ^'r ^"'""^"^ °^ errors in a s:?^^^ 

to either correct an error or to indicate that the replacement of the unit Is necessary 

n either event, this causes a portion of the fault tolerant computer to loose lock step" with the remainina 
portio^o^ the fault tolerant computer thereby reducing or eliminating the fault tolerance' onhe system ' 
«i„ni^ ^"^ 5 ! ^^"^""^^ ^ 3^ 3bove is used to delect the prosenoe of a 

AS t nln^'Jh ''"'"9 ^''^ DMAoperation ««thout losing lode st^peStan 

As mentioned above during a read of main memory if an error is detected in syndrome circuit 2Mb a^ona 

™2^1„ r or! V '^^'^ "^"^ '""'""'y ^ «'"9'« bit error a single bit error signal Is 

rillL fi 'J^ . ^ "^'"^ ^ ^'9"^' ^ '^eze the address in register 1801. That Is 

e rd^r^T/ol"" T!T ^"^ ^ DMAcontroT?ircult 134 b thaun 

otherS^e ' "'3""' '° "'^^e^Ponding Xerror circuit 138' In th^ 

A time delay is associated with the transmission of the single bit error signal to the Xerror circuit 138' of 
the other .one. While the single bit error Is being tronsmitted to the other zL. the zonrhTs ^tUed to 
proce^ instructions, and as such, the memory address at which there is a single bit error has^Zated 
throughthereg«terpipeline180a-180i. The number Of steges in the register p^ 

m , *° '"""^^'^ "^'^^ when the xl^ro^c^^ 

_^8.e.^,ves the single bit error signal, itfreezes the addr^^ 

io I J5I'^"^!L"'* "^^ ''^^^ ^ "'"9'^ ^ »a™e memory location, the zones operate 

inlocl(-step to correct the single bit error as described below on. me zones operate 

the subSTnlr Z i J "'"^'"^ '^3'*'^'" ■ ^bove. After completion of 

*esubtransfer. the saved address m register 180i is used by the DIWA control circuit 134b to initiate a read 
modify wnte operatK,n via mux 175 and register 176. It is noted that the otherzone is perform Jg anTdenS 
read modify write operation although it may not have had a memory eriw Penormmg an identical 

«c JI'^ ^'^^f"*^"^^ IS^b performs the read modify write operation at the address location with the error 

comrol circuit 134b causes the MUX184tosetect path 130atobe stored in theDMAread modify v^^^ 

ZJ^XJ^IHT ""^ -"^^ ^34b to write the c^rrectSrte Sto 

TvZe^Z' I T ^ '"^'"'"^ « bade to main me„^ 

ory at the memory location which previously had the single bit error. 

Although the present scheme saves the address of one error, it may be possible that two or more sinde 
bi toZvS a b r"!r*LT arrangelTtherSS Zd 
provkSTSM? of rat r ^llw ''''T' ? '''' "'^"^ ''"^"9 the transfer or alternatively to 

Terrnr 1^ ♦ "uffer into main memory upon detection of 

an error. As an alternative arrangement, the DMA control drouit 134b could also be programmed as a back 

essary for all of main memory and in particular, those portions of main memory whid, are infrequently ao 
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cessed. 

Referring now to Fig. 6D, the parity checker circuit 173 Is shown to include a multiplexer bank 210 fed via 
address data from address register 172 (Fig.6) and providing output data in response to select signals on line 
211 provided from DMA Control circuit 134b. The address bits are sequentially multiplexed five at a time to 

5 an exclusive OR bank here represented by exclusive OR gate 212 to generate an even or odd parity signal at 
the output thereof. The parity signal Is stored In state device 213 and presented to the output thereof via line 
213a and to a driver 217 to be fed to the other rail in the respective zone 12/12*. Correspondingly, the output 
of the state device 213 Is fed to a second state device 215 operating at here a X1 clock phase of the clock 
signals to be fed to an exclusive OR gate 214. Similarly, from the other rail Is provided a signal via buffer 218 

10 on line 218a to state device 216, operating at the X1 clock phase, the output of which Is also fed to exclusive 
OR gate 214. The result of the exclusive OR of the signals Is provided at the output 214a thereof as a parity 
error indication if the parity generated by each of the rails Is different thus providing a logic one level at the 
output thereof. If the parity signals are the same, a logic 0 is provided at the output of gate 214, and thus no 
parity error has been detected. This parity error is fed to the X clock error circuit 138 (Fig. 6A) 

IS Referring back to Fig. 6A, the interface and control circuit 60 is shown having a dock boundary denoted 
as a dotted line 141 separating portions of the bus interface and control circuit 60 between logic operating at 
the CPU clock frequency domain (Xclock) and the logic operating at an input / output (I/O) clock frequency 
domain (Pclock). The dock boundary occurs since In general the CPU clock frequency may be substantially 
different from the 1/0 dock frequency and moreover the CPU dock and the I/O dock are generally are not in 

20 phase synchronism. Both of the interface and control circuits 60, 80' for the primary rail 41 and phantom rail 
41' respectively have a dock boundary 141 ,141'. Here, the docks are aligned to permit synchronous transfers 
of data between the I/O and CPU portions of the system. The dock synchronization is provided by logic 150 
comprised of a protocol synchronism circuit 142 and X dock (CPU dock) and P dock (I/O clock) circuits 140 
and 144 respectively. 

25 Referring now to Fig. 7, a simplified block diagram showing the docking relationship between the Zone 
A (12) and zone B (12') for the fault tolerant computer system 10 is shown. In particular, it should be noted that 
in fault tolerant computer system 10 checking and communication occurs between processors in zones A and 
B. It Is necessary therefore for all subsystems in each of zones A, B to operate in a synchronous manner. That 
is each of the central processing units and associated logic in the primary and phantom rails 41,41' of Zone 

30 A as well as corresponding primary and main rails 41 ,41' of Zone B should operate with clock signals which 
are synchronized. 

However, the use of a common dock circuit to generate a dock signal for the zone A and zone B processors 
is undesirable. Here a fundamental requirement for the fault tolerant computer 10 Is that no single point of failure 
in the fault tolerant computer 1 0 should cause the entire system to shut down. To allow a single clock generator 

35 therefore, to provide docking signals for both zone A and zone B would run the risk that in the event of failure 
of a dock generator, the entire computer system 10 would be nonfunctional. 

To allow CPU and I/O speed to be optimized it is also necessary to operate each subsystem at a maximum 
rate Irrespective of the rate of the other subsystem. Nevertheless. It is necessary to maintain the ability to com- 
municate efficiently and reliably between the CPU and the 1/0 portions of each of the computers. Accordingly, 

40 it should be noted that each zone 12, 1 2' has a pair of voltage control osdilators 230/230' (for zone 12). These 
voltage control oscillators are phase locked by drcuits (not shown) to permit the primary 41 and phantoms 
41' rails in Zone A to operate in synchronism. Further details, in the operation of the clocks within a zone will 
be discussed In conjunction with Fig. 8. it should further be noted that the fault tolerant computer 10 further 
includes a voltage contrdled I/O oscillator 232 here operating at 50 MHz. This voltage control crystal osdilator 

45 (VCXO) Is phase locked to a voltage control crystal osdilator 232' in Zone B (zone 12'). 

Upon system configuration, one of the two osdilators, 232, 232' Is programmed to be a dock slave while 
the other remains a dock master. The dock master oscillates at its center, or nominal frequency. The dock 
slave phase-locks to the dock master. Oscillators 232,232* are voltage controlled crystal osdilators (VCXO's), 
having a tuning range that is relatively narrow, of typically ± 0.02%. In the event of a master oscillator failure, 

so the slave oscillator will drift in frequency, however, it will be bounded by its relatively narrow tuning range. 
The resulting small change in the period of the dock signal is Insufficient to cause disruptton of processing. 

Referring now to Fig. 8. a simplified block diagram of the single fail stop processor 1 3. generally described 
in conjunction with Fig. 2A is shown with certain details omitted concerning subsystems on the CPU unit and 
the I/O unit to permit a dearer understanding to be conveyed of the docking throughout the system 13. 

55 As described in conjunction with Fig. 2A, the single fail stop processor 13 includes a primary rail 41 and 
a phantom rail 41'. Referring in partteular to the primary rail 41, It is shown that the primary rail 41 includes 
a voltage controlled oscillator 230 here a conventional voltage controlled oscillator with a tuning range of 200 
MHz to 400 MHz. The voltage controlled oscillator 230 feeds an output signal in the above range to the 
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Z^^ ^V > : ^S^'^t'^'l 15 derives a CPU system clock SYSCLK with on device clock gen- 

S?iTl3em^?nrt 1^ 'i^^^ " NVAX^ Processor (used for processor 42 on 

CPU/Memory unit 15). Other processors would have a correspondingly similar type of arrangement The out- 
put from the processor 42. (i.e. SYSCLK2) is fed to a conventional clock driver cLit to pnSluce as^Tof 
IOC* p ases of the SYSCLK2. The clock driver circuit is typically a device such as MoLla Part n mbe 

Ih^,TTTK 'TtJIT'*' ""-^ ''^^ ^^^^ P"'-'des four clock phases 

Phases To - T3. The TO and T3 dock phases are used to provide synchronism between the various logic circuits 

ZxHn t "^'^'^ "'^^^^ TO and T3 dXhases 

The Xclod< phases are approximately 25% duty cyde dodc phases as shown in Figure 8A 

Sr^A , 1^ ^T^^ ^" ' P'^^''^^^ f^^-^ '/O 25(Fig. 2B). Here the I/O clodc 

MULKA IS fed from the crosslink communications device 65/65' of I/O unit 25(Fig 2B) 

Here a dod< frequency interface circuit is provided between the CPU dock and the I/O ckwk as will be 
toed to produce a high rate of data transfer between CPU and I/O while permitting a relatively large dif- 

SduT, , h"'h ^^'^^'^ ""^ "° ■T''^ '"t^^^^'^^ "-ntrol circuit 60 thus 

1^ u f. ^"'^''^^ ^■^"'^^^ conjunction with Fig. 8 which in conjunction 

LcicLKV^^^^^^ ?2 CPU-'-CMCLK from divide by M circuit and I/O clock signal 

l/0_LCMCLK arefed to aphase detector circuit 248. The phase detector circuit 248 as will be further discussed 

cVrS^K : '7:;''' '° '^^ - P^'-- different bLtTn^ 

ihe^^n , "^r".'"'^ lO.LCMCLK clod< signal to drive a loop amplifier 246 and produce from 
^Z^ZZ^"^TJuTl"^'''\'" *° "'^"^ o"»'e voltage controled osdilator 

J^'Jm T^,' . ^ ^'^"^'^ phase lodced to the I/O signals by changing the frequency of 
me voltage con rolled osdilator 230. Correspondingly, a SYSCLK1 signal from the NV/G(* Lvice 42 i^S to 
the Phantom rail 4 via a delay circuit 242 which here is provided by a passive lumped element delay line. 
Delay crcuit 242 which is here a five nanosecond delay is used to permit CPU device 42' to input data at 

Lo,i^!rJtV? Z"" '^'^ ""^ "'^"S ^ 5 nanosecond delay of the 

h.rflTri ! ^ propagation delay of the data on buses 43b/43b- and the propagation delay of the 

.!fK H^, - P^^^^ detector 248' provides an error or output signal m accordance 

^1 4rSr ^!lT"^uZ voltage controlled osdilator 230' in the Jhantom 

SUn, ™n H K ' ''^""^ '""^ ^'^^^ ^"t"'"*'' °««="'«tor frequency of the 

pnrT«2^ rail 41 and by inference to the voltage controlled osdilator frequency of the I/O unit 25 (Fig 2B) 

.inn^^^'T "r""!^ "'^"'y *° « ^'-^Pl'^ed timing diagram showing the relationships between the dodc 

S?siKi 1' T^T' '"S"^* °" ^•'^ "° ("^'fl- 2B) is shown. As not^ the 

ffffJr^^J^^ T . ^"^'^ P*"^"*"" '*1' '♦I' P'^^^^ '"^^^ here a five nanosecond 

1„1 « ^''^ "l""®^ P^vided by the delay circuit 242 between the rising edges of the two 

It « noted here that phases P1 and X3 are used for data transfer as will be further described below 
driv«^!L'»r'*^n" ""^t! ^ CPU signals XI and X3 are provided here by the coincidence of the CPU dodc 

Brirri%"rnt:dr^ 

I/O dtlXf pTp3. S: !^d ^""^ "° ^ ^ 

»„H ?h ' r2?«T "^'""'^ '° P*""'* «y"«*""0"s transfers between the I/O unit 25 (Fig 28) 

phantom rail 41 in both Zone A and Zone B indudes drcuits substantially identical to those shown in Fig 9 

c^rlS^IO 

J"® circuits shown in Fig. 10 are here provided on the Bus interface and contrd unit 60 (Fig. 6) and as 
bnefly mentioned in con unction with Fig. 8. indude a divide by M sedion 252 and a divide by N sect on 250 
numd ' ^^"^ '"'''"^^ " '^^'•^^ 262 which here shows a 

^„?^ 9ft?? frequency as will be mentioned below shortly. The divide by N logic further indudes a 4-bit 
counter 264 as well as a divide by two drcuit 263 which is fed via MCLKA signal 261. The 4.bit counter 264 
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provides 4-bits which are used as addresses to select one of sixteen I/O transfer (lOLCR) enables via a 16 to 
1 MUX circuit 268. The 1 6 to 1 MUX circuit 268 is provided inputs from a Input / output latency control register 
266 which is loaded a code which is detected in accordance with an algorithm as will be described below. Suf- 
fice It here to say that in accordance with the selected signals which are provided from the four bit counter 264 

5 which is in accordance with the code periodically loaded into the counter, one of the codes loaded will selec- 
tively be loaded in the I/O latency control register 266 as an output on line 269. The enable signal on line 269 
is fed to first portions 280a, 270a of state devices 280, 270, respectively. 

The CPU logic has a corresponding divide by M circuit 262a having similar circuits as described for the 
I/O logic divide by N logic 250. In particular, the CPU logic 252 has a hard loaded register store 262a which 

10 provides a number M corresponding to a remaining portion of a least common multiple ratio of the I/O dock 
and CPU clock as will be described below, and a 4-bit counter 266a which provides output bits as enables to 
a 16 to 1 MUX 268a. As with logic 250 a software, loaded CPU latency control register 266a Is fed data which 
provide enable signals as inputs to here a 1 6 to 1 multiplexer 268a. The select signals from control 266a permit 
one of the enable signals to be transferred through the multiplexer as in an enable output on line 269a. The 

IS enable signal on line 269a is fed to second portions 270b, 280b of state devices 270, 280 as shown. Thus, 
circuit 280 which is comprised of state device 280a and state device 280b Is used to effect a transfer of data 
operating at the I/O dock frequency to logic operating at the CPU dock frequency whereas circuit 270 com- 
prised of circuits 270a and 270b is used to effect a transfer of data operating at the CPU dock frequency to 
logic operating at the I/O clock frequency. Details of circuits used to provide the I/O CPU data synchronization 

20 wilt now be described with reference being made on an as needed basis to the timing diagram of Fig. 9A. 

Referring in particular to Fig. 10, the counter circuit 264 is shown comprised of a plurality of here four D 
type flip flops (not Individually referenced) as well as combinatorial logic circuits 264a comprised of AND gates 
and Exdusive-Or gates (not individually referenced). Counter 264 is here a down counter and indudes a mul- 
tiplexer bank 265 disposed between the combinatorial logb bank 264a and the D type flip-flops 264 to permit 

25 one of two paths of data to be used as inputs to the D Inputs of the D type flip-flops 264 In a normal period. 
Here each of the combinatorial logic networks are coupled to A inputs of each of the bank of multiplexers 265. 
The B inputs of each of the bank of multiplexers 265 are coupled via one of lines PCLK40 - PCLK43 from the 
hard loaded divide by N register 262. Register 262 Is fed a predetermined value corresponding to. the least 
common multiple ascertained or determined between the period of the CPU dock and the period of the I/O 

30 dock. Here this data is a hard loaded data provided from the ROM Bus 15b (Fig. 6A). Thus, this value is loaded 
Into register 262 during initial powering of the computer system 10 and in particular each of the rails, 41/41*. 
This value is fed to each of the multiplexers in the bank of multiplexers 265. Here counter 264 Is preceded by 
a divkle by two stage counter 263 since dock signal MCLKA is twice the frequency of the Pdocks (i.e. PI). 
The outputs from the divide by two stage 263 as well as counter stages 264 are coupled to a NAND gate 

35 261 to decode the state 00010 which is the state preceding the state when the counter 264 would return to 
zero. This signal at the output of gate 261 is fed to a multiplexer 261a and the output of multiplexer 261a Is 
fed to a flip-flop 261b. 

Mux 261a is also fed by a signal PH_CLK.DET indicating the decode state of the phantom rail 41*. Asignal 
DET_SEL, is also fed to the MUX 261a. DET_SEL is provided from a status register (not shown) in the primary 

40 rail or as a hardwired signal. As mentioned above a similar docking arrangement is provided in the phantom 
rail. Thus an equivalent Mux (not shown) is also fed by a signal which Indicates the decode state of the primary 
rail 41 and the select signal DET_SEL Signal DET.SEL is used to always select the primary rail signal in both 
the primary rail and the phantom rail. This insures that the loading of the counter 263 and a corresponding 
counter (not shown) in the phantom rail 41 are always being loaded with the predetermined starting state at 

45 the same time. This insures lock step operation between the two rails of the zone 12. A similar arrangement 
is provide for zone B. 

The output of flip-flop of 261 b changes to the value of the output of gate 261c when that counter is at state 
00000 and is used to provide a select signal to control the multiplexer bank 265. Thus the signal for flip-flop 
261b permits either the combinatorial logic network 264a or alternatively the register 262 to provide the outputs 

50 from the multiplexer 265 as the corresponding inputs to the D flip-flops of divide by 2 circuits 263 and counter 
264 for the next dock cyde. In this manner, the down counter is permitted to reload the initial value of the count 
sequence or the least common multiple in accordance with the dock frequencies from the register 262. 

Referring now to Fig. 11 the transfer and enable logic 266 is shown to include an I/O control register 267 
here comprised of a pair of register stages 267/267' which here each stage having a pair of registers used to 

55 feed here sixteen data bits from Bus 130 into one of a pair of 8 to 1 multiplexers 268. Multiplexers 268, have 
outputs to which are coupled to a 2 to 1 multiplexer 268' as shown. Dock signals PLCK2Q1H - PLCK2Q4H 
are used as selects for the multiplexers 268/268' respectively as shown. The output of multiplexer 268' is fed 
to here a five stage pipe delay 269 with the three odd number stages being docked at the frequency signal 
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PCLK3P3 and the two intermediate stages clocked at tlie frequency of signal PCLK3P1. Here the last stage 
of the pipe delay 269 Is a stage of four flip-flops used to provide sufficient dock signals for the remainder of 
the circulL 

Referring now to Fig. 12 a recirculating state device pair circuit 270 used to effect transfers from the I/O 
logic to the CPU logic is shown to include a first state device 270a comprised of here a JK flip-flop 272 which 
has a first input of P3 data that is the data at the I/O dock frequency and which is clocked at the frequency of 

the P3clock.lt should be noted that the circuit 270 is replicated as many times as Is necessary to accommodate 
thewidthof Bus 15d. 

The P3 data is applied to the J and K Inputs of flip-flop 272a. The output of flip-flop 272a provides the 
first input of gate 273a whereas the second input of exdusive-OR gate 273a is provided from the output of a 
second JK flip-flop 278. The P3 data is also provided as an input to an OR gate 275a and in the second input 
of OR gate 275a is provided from the output of the exdusive-OR gate 273a. The output of OR gate 275a is 
provided as an input to an AND gate 277a with the second input to AND gate 277a provided by signal P3XFREN 
which IS the transfer enable for the P3 dock provided from signal 269 (Fig. 9). The Q output of flip-flop 278 
provides the input to a second state device 270b. The second recirculating state device 270b includes a mul- 
tiplexer 272b and a pair of D flip-flops 274b. 276b which are docked at the X3CLK dodk frequency. The output 
of the last flip-flop 276b. provWes the first input to an exclusive-OR gate 279 whereas the outputf rom the first 
flip-flop 274b provides the second input to exdusive OR gate 279 as well as the A input to the multiplexer 272b 
The select input to multiplexer 272b is provided from signal X1 transfer enable (X1XFREN) provided from an 
enable circuit similar to that as will described in conjunction with Fig. 12b for the X dock signal. 

Referring now to Fig. 12A, a second recirculating state device pair of circuit 280 used to effect transfers 
from the CPU dock logic to the I/O dock logic is shown to indude first state device 280a comprised of here a 
D flip-flop 282 which has a first input of X3 data in that is clocked at the CPU do* X1. The output of D flip- 
flop 282 is fed to a second flip-flop 283 here a JK flip-flop and is fed to both J and K inputs of said flip-flop 
The output of device 282 is also fed to a OR gate 284b whereas the Q not output of JK flip-flop 283 is fed to 
a first Exdusive-OR gate 284a. Output of Exdusive-OR gate 284a provides a second input to OR gate 284b 
The output of OR gate 284b is fed to an AND gate 285 which has a second input X3XFREN enable which is 
the transfer enablefrom line 269a. The output of AND gate 285fbeds a second JK flip-flop 286 as shown The 
output of JK flip-flop 286 feeds a first input of a two to one iVIUX 287. Aselect input from MUX 287 is fed signal 
P3XFREN whteh Is the transferred enable signal from line 269. The output of MUX 287 is fed to a first D flip- 
flop 288a which in turn is fed to a second D flip-flop 288b. D flip-flop 288a Is clodded at the P1 dock phase 
whereas D flip-flop 288b is dodged at the P3 dodk phase. The output of the P3 dock signal is fed bad( to the 
input of MUX 287 whereas output from gate 288b is also fed to the input of a third D flip-flop 288c. The output 
of flip-flop 288c and flip-flop 288b are fed to an Exdusive-OR gate 289. Output of Exdusive-OR gate 289 is 
output data transferred or synchronized to the P3 dock of the I/O unit 25. Thus, state device 280 is used to 
effect data transfers between the I/O and CPU from the CPU to the I/O whereas state device 270 ( Fig 12A) 
is used to effect data transfers from the I/O to the CPU. 

A brief discussion of the operatton of circuit 270 will suffice also for a discusston of circuit 280 and will 
be useful in understanding this aspect of the inventton. Referring bad^ to Fig. 12A, when P3 data is in a high 
stats fbr one PI CLK cyde. and If P3XFREN is also high, indicating that a transfer can occur, the P3 data will 
pass through gate 277a to JK flip-flop 278. On the next P1 cycle the 'V at the output of gate 277a will toggle 
flip flop 278. If X1XFREN is also 'V, then on the next X3 CLK cyde flip-flop 274b will change state and its 
stale will be different from the state of f lip-ftep 276b for one dock cyde, thus creating a high level at exdusive- 
or gate 279. Thus the drcuit provides a pulse, synchronized at a first dock signal, which is converted to a state 
change and which is reconverted to a pulse which is synchronized to a second dock signal. If when P3 data 
was high P3XFREN is not enabled, then flip-flop 278 does not toggle. However, flip-flop 272a does toggle 
Eventually when P3XFREN Is at k>gic 'V level, gate 273a detects a difference in logic stetes between^te 
device 278 and 272a, provides a logic 'V to gate 275a, and thus onto gate 277a permitting JK flip-flop 278 to 
toggle. The transfer is complete as discussed above. If X1XFREN is not in a logic 'V state when the state of 
278 toggles, then the state of flip-flop 278 and flip-flop 274b will be different When X1XFREN is at logic 'V flip- 
flop 274b will change to the state of flip-flop 278, thus completing the transaction. 

Referring now to Fig. 12C. a dreuit 291 to transfer data signals which are levels (ie. more than a couple 
of dock cydes in width) rather than pulses as described above, is shown to indude a MUX 292 having as a 
first input P3_Date_ln and as a second input, an output from a state device 294 which has its input coupted 
to the output of the MUX 292. A select input to the MUX 292 is fed via a transfer enable signal from the mul- 
tiplexer bank 268 (Fig. 11). This signal is used to have the MUX 292 select the output from the state devtee 
294 to be fed to the state device unless the signal P3_XFR_Enable indicates that data on line P3_DataJn can 
be transfered at the next dock signal P3_CLK. Data is transfered acnjss the dod< boundary 295 to a second 

24 



EP0 606 779 A2 



MUX 296. A second input of the MUX 292 is fed from an output of a state device 298 which has its input coupled 
to the output of the MUX 296. A select input to the MUX 296 is fed via a transfer enable signal from the mul- 
tiplexer bank 268a (Fig. 9). This signal is used to have the MUX 296 select the output from the state device 
298 to be fed to the state device 298 unless the signal X3_XFR_Enable indicates that data from the state device 
5 294 can be transfer to the state device 298 at the next clock period of X3_CLK. This circuit is simpler than the 
ones mentioned above and thus can be used to transfer levels. As with the pulse circuits, this circuit is repli- 
cated as many times as Is need to transfer the full width (data, control, and addrress) of data signals. 

Referring now to Fig. 13 a circuit 300 used to provide a output signal to a phase detector 248 (FIG. 6) and 
as will be described in more detail in conjunction with Fig. 14 is shown. In order for phase detector 248 to op- 
to erate optimally it is desirable to provide a output signal having a fifty percent duty cycle. However, since the 
clocking throughout the CPU as well as the I/O logic 252. 250 respectively uses several clock phases having 
pulse widths which are less than fifty percent duty cycles, circuit 300 is provided to transform the dock signal 
into a fifty percent duty cycle signal. The circuit 300 is shown to include a bank 310 of exclusive-OR gates. 
Here, the exclusive-OR gates have a first input fed by signals XCLK2Q0H - XCLK3H the CPU clock, clock 
15 divider state signals, and each have second inputs fed by signals XLK4X1 H - XLK4X3H the CPU clock divide 
value signal. A fourth exclusive-OR gate of the bank of gates 31 0 has the second input provided as the logical 
NOR of signals XCLK4X0H - XCLK4X3H respectively. That is the Q outputs from the 4-blt divide by M counter 
264 (Fig. 10) are exclusive-OR together with corresponding bits from the control word register 262 ( Fig. 10). 
The outputs of these exclusive-OR gates 310 are NOREDed together via NOR gate 312. The output of NOR 
20 gate 312 provides the K input to the first JK flip-flop 314 with the J input of JK flip-flop 314 provide from a 
signal XCLK2LCMR (Fig. 11). JK flip-flop 314 is clocked at the clock frequency of XCLKI^TO (approximately 
20.8 MHz). The output of flip-flop 314 is fed to a pair of flip flops 316a, 316b here each D type flip-flops with 
flip-flop 31 6a clocked at the inverse phase (via an inverter, not referenced) of clock XCLK1_T0_L and flip-flop 
316b docked at the signal XCLK1_T0_H. The outputs of each of these two flip-flops are fed to a pair of AND 
25 gates 318a. 318b as shown and the outputs of AND gates 318b provides signal XCLK2_CPULCM and the out- 
put of NAND gate 318a is fed to an inverted 319 to provide signal XCLK2_L0CAL-CPULCM. 

The fifty percent duty cycle circuit 300, which is used in the divide by M logic of the CPU\Memory unit 16 
operates as follows: 

The JK flip-flop 314 is set on the positive dock edge following the dock which reloads counter 264a with 

30 the divide by M value. On the negative-going edge of the aforementioned dock cyde, the setting of JK flip- 
flop 314 sets D flip-flop 316a. On the next positive clock edge, flip-flop 316b Is set. The outputs of flip-flops 
316a and 316b are ANDed together by AND gate 318 to provide the 50% duty cyde output A decoder, provided 
by logic drcuit 310 provides an output signal that decodes the M/2 state of the counter 264a. This signal is 
applied to the K input of JK flip-flop 314 to reset JK flip flop 314 on the next positive clock cyde. Resetting 

35 JK flip-flop 314 likewise resets D flip-flop 316a on the next negative dock cyde and resets flip-flop 316b on 
the subsequent positive dock cyde, thus truncating a half-period from the output provided at gate 31 6. If the 
divide by M is an even number, a reset input on JKf lip flop 314 is held continuously In the reset state to maintain 
the 50% duty cyde. Thus the circuit can be used to provide 50% duty cydes for both even and odd values of M. 
Referring now to Figure 14, a phased locked loop 350 to facilitate data transfer between the CPU\Memory 

40 unit 15 and the 1\0 unit 25 includes a divide by M circuit 252 which is coupled to the CPU dock TO (indirectly 
sourced by the VCO 230 and CPU 42 as described previously) to provide a 50% duty cycle dock signal 
CPU^LCMCLK (Figure 13) to the phase detedor drcuit 248 via line 252b. In addition, the I/O dock MCLKA is 
modified by the divide by N circuit 250 to provide a 50% duty cyde signal lO^LCMCLK on line 250b to the 
phase detector circuit 248 as generally mentioned above in conjunction with Figure 9. 

45 Because the CPU\Memory unit clock TO cydes at a frequency which may be different from the I/O unit 

dock MCLKA, a least common multiple (LCM) of the two clocks is calculated and fed to the divkje circuits 250 
and 252 as described herein previously. For example, the CPUXMemory unit dock TO may cyde at 40 ns while 
the I/O may cyde at 50ns. The lowest common cyde time between the two docks would be 200 ns. Therefore, 
the LCM fed to the divide by M circuit is 5. while the LCM fed to the divide by N circuit is 4. Therefore, the 

50 phase detector circuit 248 receives two dock signals, lO.LCMCLK on line 250b and CPU.LCMCLK on line 
252b. both of which have a cyde time of 200 ns, but which may be out of phase. 

The phase detector 248, as mentioned previously, provides a signal which is a measure of the phase error 
between the CPU dock and the I/O dock to drive a loop amplifier 246 and produce from the loop amplifier an 
output signal which is used to adjust the frequency of the voltage controlled crystal oscillator (VCXO) 230. 

55 The VCO 230, as described previously, is an oscillator having a frequency which may be varied by chang- 
ing the voltage level of an input control voltage signal. The VCO may be a simple inductor/capacitor pair con- 
figuration or a Crystal oscillator, as well as other variations known to those skilled in the arL Consequently, 
the output frequency of the output clock signal from VCO 230, and thereby the integrity of the data transfer 
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between the CPU\Memory unit 15 and the I/O unit 25. is largely controlled by the phase detector 248. 

Referring now to Figures 15A. 15B, and 15C. typical transfer curves for phase detectors illustrate the re- 
lationship between the phase error*, and the resulting output voltage provided by the phase detector, where 
Vp is the peak voltage (or saturation voltage) supplied by the phase detector. The transfer curve for a three 
state phase detector Is shown in Figure 15A. The detection range of the three state phase detector Includes 
values between plus or minus 1%, providing a detection slope of Vp/2n. Hence, any phase error <|> is related to 
a unique voltage level over a 4n range. 

Referring now to Figure 15B, an example of a transfer curve of a circuit which combines the use of two 
three state phase detectors with their respective outputs ored together, where each triggers on an opposite 
edge of the clocks being phased locked, is shown. The dashed lines indicate the respective transfer curves 
for each separate three phase state detector circuit, while the solid lines indicate the sum of the pair of three 
state phase detector circuits, providing a transfer curve for the dual, three phase detector circuit. Given the 
characteristics of the dual three-state phase detector circuit, with the outputs ored together, a problem may 
occur where the phase error value (|> is zero for values of plus and minus ji. Hence, a circuit using dual three 
state phase detectors may have the potential of locking the clock signals exactly % radians out of phase. 

The present invention inhibits the problem of the CPU clock and 10 clock locking out of phase by providing 
a circuit which recognizes that the two clocks are exact % radians out of phase and precludes a phase detection 
circuit from indicating that there is no phase error. The voltage vs. phase error transfer curve of a circuit in 
accordance with the present invention is illustrated in Figure 15C. As shown, circuit correctly provides no vol- 
tage compensation when the phase error is 0. In addition, the transfer curve illustrates that the present in- 
vention permits the phase detector circuit 248 to provide its peak output voltage to compensate for any phase 
error over « radians, and has double the slope of the conventional three state phase detector for phase errors 
between and n radians. The increased slope (increased gain) of the transfer curves Indteates that the phase 
detection circuit 248 provides more compensation voltage per degree of phase error difference, thereby re- 
ducing the static error by a factor of two. Offsets resulting from the loop component characteristics are mini- 
mized by the increased gain of the phase detection circuit 248. 

Referring now to Figure 1 6, one embodiment of the phase detector 248 includes 4 state devices 320, 321 , 
322 and 323 which are used in conjunction with logic gates to detect rising and falling edges of the input docks 
lO.LCMCLK on line 250a and CPU_LGMCLK on line 252b in the following manner. All four state devices, 320, 
321, 322 and 323 (which may be typical D-type flip flops known to one skilled in the art) have the Data input 
pin tied "high", or to a logical '1' value in this embodiment State devices 320 and 321 are respectively clocked 
by the rising edges of CPU_LCMCLK and IO_LCMCLK. State devices 322 and 323 are respectively clocked 
by the falling edges of CPU_LCMCLK and IO_LCMCLK via Inverters 330 and 331 . 

The output of both state devices 320 and 322 (Qa and Qc respectively) are ored together by logical OR 
gate 320d to provide a signal VCO.Detect on line 248a. Thus, the signal VCO_Detect will be asserted when- 
ever the phase detection circuit 248 detects a rising or falling edge of the CPU_LCMCLK. Similarly, the output 
of state devices 321 and 323 (Qb and Qd respectively) are ored together by logical OR gate 321 d to provide 
a signal REF_Detect on line 248b. Thus, the signal REF.Detect Is asserted whenever the phase detectbn 
circuit 248 detects a rising or falling edge on the lO^LCMCLK (the reference dock In the phase lodced loop 
as described in conjunction with Fig. 14A). 

Typically, in three state phase detectors, the VCO_Detect and REF.Detect signals are reset when both 
clodcing signals being compared are in the same logical state. However, in the present invention, the arrange- 
ment of logic components. 321a, 321b. and 321c and logic components 320a, 320b, and 320c preclude the 
phase detection circuit from locking the CPU_LCMCLK signal to the IO_LCMCLK signal when they are out of 
phase by inhibiting the signals VCO.Detect and REF.Detectfrom being reset in certain predetermined states 
even though both CPU_LCMCLK and IO_,LCMCLK may be asserted. 

Referring now to Figure 17A, an example of an Instance wherein the CPU^LCMCLK and IO_LCMLCK are 
operating In phase is shown. When the CPU.LCMCLK Is asserted, the Qa output from state device 320 also 
Is asserted (after a time period equal to the dock -> data propagation time of state device 320) If as shown 
here. lO.LCMCLK is in phase lode with CPU_LCMCLK. then Qb also is asserted after a time period equal to 
the dod( -> data propagation time of the state device 322). Qc and Qd are here deasserted as shown. When 
both Qa and Qb are in an asserted state, an AND gate 320e feeds a '1' value to each of the NAND gates 320a 
and 321a. In essence the outputs from delay gates 320b and 321b are enable signals, which enable the reset 
signal from gate 320e to pass through gate 320a, and thus reset the state devices 320 and 321 . Because both 
Qc and Qd are deasserted, NAND gates 321 c and 320c provide a '1 ' value to delay gates 320b and 321 b which 
in turn provides a '1' value to the NAND gates 320a and 321a, thereby enabling the '1' value souroed from 
gate 320e to provide a reset signal to the state devices 320 and 321 . This reset signal is fed to asynchronous 
dear inpute of stete device 320, 321 whfeh In turn cleare the stete devices 320, 321 . Thus, Qa and Qb remain 
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asserted for a period of time (tp1) equal to tlie propagation delay through the string of combinatorial gates 
320a and 320e. at which point they deassert as shown. 

Similarly, when the CPU.LCMCLK Is deasserted, the Qc output from state device 322 Is asserted (after 
a time period equal to the clock -> data propagation time of state device 322). If, as shown here, IO_LCMCLK 

5 Is in phase lock with CPU_LCMCLK. then Qd also is asserted after a time period equal to the clock data 
propagation time of the state device 323). When both Qc and Qd are in an asserted state, the assertion of 
both Inputs at gate 322a provides a '0' output, to the asynchronous dear pin of state devices 322, 323 to dear 
the state devices. Thus, Qc and Qd remain asserted for a period of time proportional to the propagation delay 
through the NAND gate 322a and the flip flops 322 or 323. 

10 Referring now to Figure 1 7B, an example of an instance in which the CPU_LCMCLK leads the IO_LCMCLK 
is shown. When CPU_LCMCLK is asserted, the Qa output becomes asserted. The Qa output remains asserted 
until the IO_LCMCLK signal becomes asserted, at which point the Qb signal also becomes asserted. As de- 
scribed previously, the output from NAND gates 320c and 321 c feed an enable signal to delay gates 320b and 
321b respectively. This enable signal allows the reset signal from gate 320e to propagate through gates 320a 

15 and 321a, and reset the state devtees 320 and 321 . In Figure 17B, when the lO^LCMCLK signal becomes as- 
serted, the Qb output becomes asserted, thus creating a reset signal at gate 320e. However, during the same 
perbd, as CPU_LCMCLK falls, the Qc input becomes asserted. Because both Qb and Qc are asserted, the 
reset enable signal provided by gate 320c is disabled. The propagation delay of delay gate 320b must be suf- 
ficient delay the output of gate 320c from disabling the enable signal In order to allow the reset signal from 

20 gate 320e to pass through gate 320a and reset state device 320. Therefore, the delay gate should provide a 
delay at least equivalent to the Clear-Q propagation of the state device plus the delay of gates 320a and 320e. 

As shown in Figure 17B, when the CPU_LCMCLK leads the IO_LCMCLK by 180 degrees, the resulting 
VCO detect signal is a constant high asserted signal, while the resulting 10 detect signal is a train of pulses. 
The output, therefore, of the pulse detector is the differential voltage between the two signals, which will pro- 

25 vide the maximum output voltage In order to remedy the phase error. 

Conversely, referring now to Figure 17C, an example of the output of a design using the dual three state 
phase detector implementation, as discussed with reference to Figure 15B is shown. An example wherein 
CPU.LCMCLK leads the lO.LCMCLK by 180 degrees illustrates that the final output signals VCO detect and 
VREF detect are mirror image pulses, and as such, the differential voltage of the signals is 0. Thus, the phase 

30 error between the signals would not be corrected through the use of a dual three state phase detector. 

The phase detector of the present invention thereby improves the phase locking process by detecting a 
phase difference for ix>th the rising and falling edges of the respective clocks. This provides improved tracking 
of the VCO frequency (and subsequently the CPU\Memory unit dock frequency) to the reference (I/O unit) 
frequency. In addition, because the output pulses from the phase detector are double the frequency of con- 

35 ventional phase detectors, the low pass filtering of the reference frequency is facilitated due to the relative 
ease of filtering higher frequency signals as opposed to filtering lower frequency signals. Finally, the present 
inventbn provides a frequency transfer curve having double the slope of the previous three state phase de- 
tectors, thus enabling increased voltage compensatton to correct phase errors, and thereby reducing static 
phase error in the phase-locking process. 

40 IHaving described preferred embodiments of the invention. It will now become apparent to one of ordinary 
skill in the art that other embodiments incorporating their concepts may be used. It is felt, therefore, that these 
embodiments should not be limited to the disclosed embodiments. 



45 Claims 

1. Apparatus for handling docking signals which have a rising edge and a falling edge, comprising: first 
means (X dock) for providing a first docking signal (X3): second means (P clock), responsive to an error 
signal, for providing a second clocking signal (PI); means (248, 248'), responsive to a rising edge and a 

so falling edge the first and second docking signals, for providing the saki error signal. 

2. Apparatus as claimed in claim 1, wherein the means for providing the error signal further comprise: first 
detection means, responsive to the first docking signal, for providing a first signal indicating a state tran- 
sition of the first clocking signal; second detection means, responsive to the second docking signal, for 

55 provkiing a second signal indicating a state transition of the second clocking signal. 

3. Apparatus as daimed In daim 2. wherein the first detection means further comprise: means for deas- 
serting the first signal and the second signal when the first signal is asserted and the second signal is 
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4. 



7. 



8. 



9. 



10. 



asserted wherein the means for deasserting comprise: means for precluding deassertion of the first sig- 
nal and the second signal when the first signal and the second signal are both asserted due to opposite 
edges of the first and the second docking signals. 

Apparatus as claimed in claim 2 or 3. wherein the first detection means further comprise: first edge sens- 
ing means for providing an asserted signal indicating a rising edge of the first clocking signal; and second 
edge sensing means for providing an asserted signal indicating a falling edge of the first clocking signal 
wherein he second detection means further comprise: third edge sensing means for providing an avert- 
ed signal indicating a rising edge of the second clocking signal; and fourth edge sensing means for pro- 
viding an asserted signal indicating a falling edge of the second clocking signal. 

Apparatus as claimed in daim 4. Including means for provkJIng a first reset signal when the first edge 

sensingmeans indicate a rising edge of the first clocking signaland the third edge sensing means indirate 
a rising edge of the second clocking signal. -raMiuiuaie 

Apparatus as claimed in claim 5. further comprising means for allowing the reset signal to reset the first 
edge sensing means when the signal provided by the second edge sensing means and the signal provided 
by the fourth edge sensing means are in a predetermined state. 

Apparatus as claimed in claim 6, wherein the predetermined state is when the signal provided by the sec- 
ond edge sensing means asserted and the signal provided by the fourth edge sensing means asserted. 

il^Silf ?K !lf 'T" ^"^ °^ "^^"^ ^' ® ^""^ ^' '"^'^^'^ comprising means for allowing the reset signal 
signal provided by the fourth edge sensing means are in a predetermined state. 
Apparatus as claimed In claim 8, wherein the predetermined stete is when the signal provWed by the sec 
wrted "^"^ ^^'^^ ^"'^ ^^"^ ^"^'"9 "^'^ '8 ^ 

Apparatus as daimed in any of claims 4 to 9. induding means for resetting the second edge sensing 
^® ""^""^ ^'^Se sensing means indicate a falling edge of 

the fret docking signal and the fourth edge sensing means indicate a falling edge of the second doSdng 
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@ The apparatus includes a first means for pro- 
viding a first docking signal, a second means 
for providing a second docking signal and 
means for providing an error signal responsive 
to an offset between edges of the first and 
second docking signals. The error signal provi- 
ding means is a phase comparator responsive 
to both the rising edges and the falling edges of 
each of the first and second docking signals. It 
comprises first and third edge sensing means 
(320,321) for respectively sensing rising edges 
of tiie first and second docking signals and 
second and fourth edge sensing means 
(322.323) for respectively sensing falling edges 
of the first and second docking signals. 

The edge sensing means may comprise D- 
Type flip-flops. A first reset signal is generated 
in response to the detection that both output 
signals of the first and third edge sensing 
means are asserted. A second reset signal is 
generated in response to the detection that 
both output signals of the second and fourth 
edge sensing means are asserted. 
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